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OBJECTIVES OF THE HANDBOOK 



1 TO HELP THE READERS TO ACQUIRE THE' SKILLS' OF 

(a) stating educational objectives in behavioural terms:, 

(b) planning a curriculum and selecting learning activities: 

(c) constructing tests another measurement devices. 



2 These skills will be based upon measurable gain of knowledge 
Concerning: Q 

(a) relationship between professional training programmes and 
subsequent practice: . 

(b) principles of learning and role of the teacher: 

(c) role of objectives in educational planning: 

(d) principles and methods of curriculum planning: 

(e) principles and practices of educational evaluation. 



__.XL.aiLI E NTS 




0 



0 



D 



0 



0 



* 

Page 

CHAPTER 1 EDUCATIONAL OBJECTIVES. 

-What is meant by educational cbjectiv i 105 

-Selection of goals * - Ill ; 

-Types of educational objectives 119 / 

-Institutional objectives. » 123/ 

-Intermediate objectives 1JJ' 

-Specific objectives . c 1J9 

-Taxonomy of intellectual processes 151 

CHAPTER 2 PLANNING OF TEACHING/LEARNING ACTIVITIES j 

-The purpose of teaching /205 

-A few words about learning ..7.213 

-Teaching techniques and media ./.219 

-Curriculum planning < ../... 228 

-Ihe concept of integration ../...2J2 

-A learner oriented curriculum with built-in relevance./.,.. 245 

-Curriculum specification taoles j 255 

-What it is all about; a synthesis *. .y/ 259 

CHAPTER' 3 EVALUATION j 

-What is evaluation. .\ / JOJ 

1 -Role of taxonomies in evaluation....* / 31 4 

-Formative and summative evaluation / 316 

-Aims of evaluation , y 318 

-Methodology of evaluation ♦ / ,323 

-Attitude measurement J26 

-Evaluation by the students / 329 

-Validity, Reliability, Objectivity and Relevance 335 

CHAPTER 4 TESTS AND MEASUREMENT TECHNIQUES 

-Qualities of a test.,.. <. , 404 

-Stages "of assessment 407 

-Common defects of examinations 413 

-Types of written examinations 419 

-The programmed examination 442 

-Test construction specification tables ' 455 

-Relative and absolute criteria tests 457 

-How to calculate the Minimum Pass Level 458 

-How to do' an Item analysis 465 

CHAPTER 5 HOW TO ORGANIZE A SHORT , 501 

EDUCATIONAL WORKSHOP 

INDEX/GLOSSARY , 601 

BIBLIOGRAPHY 615 

f THERE IS AL SO A PRE- TEST on page 10 and a POST-TEST on page 613 

* Tjie pages have been numbered in such a way that each chapter always 
commences with page 01 (e.g., page 301 is the first page of the third 
chapter) whatever the number of pages in the previous chapter. 
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Preface to the fourth edition 



This fourth edition (1977) is a reprint of the 1976 e d ition with minor ed itorial 

changes* It was a collective work which took into account the critical comments and 
suggestions of numerous users of the first editions. ~ 

The first edition was prepared on the basis of documents distributed during a workshop 
organized by the Regional Office for Africa, Brazzaville, in December 1969 with the 
assistance of consultants in medical education. 1 It was revised in May 1971 and has been 
widely used in the Region and served as a basic document for many educational planning 
workshops. / 

It then became evident that it would be useful to give the reader better guidance during 
his progression through the whole educational planning cycle. Although this objective 
would cost likely be achieved by a completely programmed presentation it seemed^ that the com- 
plexity .of the problems involved was not great enough to justify a method which would be so 
intricate and long to prepare. .Only practice could say whether this was correct. In 1975 
a fully revised text was field tested and led to the 1976 revision distributed in over 
10 000 copies in four languages . 

A list of objectives ( yellow pages ) is indicated^ for each chapter so as to give the 
reader an idea of what lie can learn from it. JEo help him discover whether he has increased 
his knowledge, exercises are included" througho.ut _the Jlajn^b^Jk;^^a-pages-X^ 

The theoretical content takes the form of original documents or short excerpts ( white 
pages ) from works included in the list of references (page 615). / 



Certain pages are printed in large characters so that they can easily be made into 
transparencies for overhead projections (for technique, see page 218). 

This new edition will, in turn, be revised in the light of suggestions we hope to receive 
from users. If you would like information or explanations, do not hesitate to write to: 



0 



J. -J. Guilbert, M.D., Ph.D. (Educ.) 

Chief, Educational Planning 

Division of Heal'th Manpower Development 

World Health Organization 

1211 Geneva 27 

Switzerland 



To order copies please see list of addresses on back cover. 



Dr S. Abrahamson and Or H. Peterson (USC, DepV Med, Educ, Los Angeles). 
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FOR WHOM 

is this Educational Handbook intended ?. 

- < 

Originally, in 1969, the first edition was written for teachers of the hea It nesciences* 

- - - . • *> 

.Subsequently, however, the Handbook, was used above cxll by hundreds of participants in meetings 

organized by, or with the assistance of, WHO (educational planning workshops, teaching 

♦ 

methodology seminars ,_ etc . . • 

A recent survey covering 200 users revealed the following opinions. ^ 
A very small minority (10%) felt that the Handbook shcald be supplied only to partici- 
pants in controlled educational activities (workshops, seminars, courses, etc.) or reserved 

•a • 

for "teachers of teachers". 

The majority felt, however, that dissemination should be as wj.de as possible , and be 

/ 

directed to all teachers of health sciences at all levels, health administrators with staff 
supervision responsibilities, and to students , so as to " them to draw the maximum benefit 
from their learning activities and help them to participate in their organization. 

Although it appears To be proved that the use of the Handbook is made easier by partici- 
pation in an introductory meeting, nevertheless a motivated individual reader would seem toj 
be able to derive great advantage from it. 

★ 



THIS HANDBOOK'S PRESENTATION IS UNUSUAL . ONE WEEK BEFORE 
GOING TO PRESS I WAS ADVISED BY A DISTINGUISHED EDUCATOR 
WHOM I TRUST TO SAY A FEW WORDS ABOUT THIS UNUSUAL CONS- 
TRUCTION. - 1 TRIED. ; . . .AND DID 'NOT FIND THOSE WORDS 

THIS UNUSUAL PRESENTATION MAY IRRITATE YOU AND COMPLICATE 
YOliR WORK UNTIL YOU GET USED TO THE WAY THIS HANDBOOK HAS. 
BEEN POT TOGETHER, THERE MUST BE A REASON FOR IT... EVEN IF I 
DO NOT KNOW IT.... J.-J.G. 



* 
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INTRODUCTION 



Educational problems have been a source of concern for many years to those training health 
personnel and the alarm has often been sounded. However, the strength of the traditions 
impeding the necessary reforms has been such that it_has not been possible to avoid serious 
disorders in numerous universities throughout the world, disorders which have always been 
caused by a reaction in face of the apparent diehard conservatism of the system. 

It would, however, be negative and dangerous merely to accuse of incompetence those at 
.present in positions of teaching responsibility. They should be offered aid. 

Societies change and ha^e always been changing, but until the present century their 
evolution was relatively slow and adaptation to change was possible without unduly violent 
disorders . 

However, the form of teaching has for centuries past been a repetition of itself. The 
university has wrapped itself in its privileges and remained deaf to the cry from without. 
The needs of society, the practical side of the matter, have been left to chance. The 
specific features of the situation in each country are changing ever more rapidly, but 
higherto, unfortunately, little or no account has been taken of those features and the _ 
training of health personnel has copied the traditional systems utilized in the past. What 
is now required is tp make sure that educational programmes are relevant. 

Ther » can be no question of continuing to copy the models of the past or, in the case of 
developing countries, foreign models* 

The educational system ijeading to the development of health personnel, at all levels , 
must be re-examined within the context of the needs of the country concerned .! 

No educational system c/n be effective unless its purposes are clearly defined. The 
members of the health team must b|> trained specifically for the tasks they will have to 
perform , taking into account the circumstances under which they will work. 

These tasks can only be defined in accordance with a plm in which the nature of the 
services to be provided is specified, priorities are allotted, the staff needed to provide 
these -services determined, etc.... 

Professional training programmes must then be tailored to meet these need3 (see diagram, 
page 114;. 

There is room for some degree of optimism in this sphere, for in order to move in the 
right direction, it is not necessary to obtain financial assistance but only to use a resource 
distributed in more or less equal quantity throughout the world: mental ability. The 
management of that resource is the art of organizing talent and of coping intelligently with 
change. ' * 



* The study of needs, organization of health services, and definition of tasks and 
functions are, however, not dealt with in this Handbook. Consequently, specialized texts 
should be consulted concerning those aspects (see references, p. 615). 
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A sum of knowledge exists in the matter of management, planning and education which could 
"assist the teaching staff of existing or future health science centres as well as health 
administrators with training responsibilities. 

Defining the educational objectives of training centres is of crucial importance. 

~Thus an educational programme , instead of being the result of a non-sel xtive accumulation 

of knowledge built up over the centuries, must be shaped selectively in terms of the goal to 

be achieved. If that goal is modified in the course of time, the programme, too, must be 
modified accordingly. 



The educational objectives Must proceed from a prospective study of requirements and 
possibilities and be based on >tne definition of the tasks the various types of personnel will 
have to carry out during th&Lr professions 1 career in a given type of health service. 

If we stress the importance of^the prior definition of health activities (services), it 
is because it is a prerequisite fcr any definition of educational objectives (see diagram, 
p. 114). Indeed, if educational objectives are based on faulty principles, then the "best" 
system of training may well give "bad" results, for there is a danger that a "bad" message 
will be "better" communicated, and this is certainly not the goal sought. ^/ 

Educational objectives should be defined by using behavioural terras corresponding to the 
t _ ks to be ^accomplished. In other words, the definition must indicate what the graduates of 
a given school will be able to DO at the end of their period of education or training that they 
were not able to do before . |5y means of an evaluation system it can be determined whether 
these objectives have been achieved. 

Even though the idea of clearly defining the objectives of a professional training 
programme seems to be a simple one, it is, in fact, a ratner difficult task.- 

If the teaching staff are given an opportunity to gain the new knowledge they need and to 
acquire the appropriate modern teaching skills, they will feel mone secure and, instead of 
being confined to limited personal experience, they will accept the use of more formal 
educational research methods. 

Such findings can be powerful stimuli to institutional change, particularly when used by 
faculty members whose experience in the educational process has already alerted them to the 
ways in which educational innovation can be accomplished with the largest measure of faculty 
enthusiasm and the smallest measui? of faculty hostility. Such innovation, based upon 
carefully gathered information and developed on sound educational principles, may allow some 
of the medical education institutions to explore, in particular, non-traditional means of 
preparing the members of the health team for the professional task they must undertake. 
Without the incrusted educational tradition that long adherence to a single system creates, 
the opportunity for innovative experimentation is far greater. £ 

This is a very difficult task which may well have daunted the most conscientious. We 
consider that assistance in this field should be given to teachers. 

That is the main reason wh> this Handbook has been prepared and used during workshops on 
educational planning. 

★ ★ ★ 
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It Should be stressed at this point that the approaches suggested 



in thai 



the result of, a ^deliberate choice by the author and reflect his ideas in the field of educa- 
tion. Various theories sometimes regarded as- contradictory are explained to the reader so 
as to give him .food for reflection, rather ; than to subject him to any philosophical con- 
straint. It is for the reader to make his choice, to draw initial conclusions; and«, in 
particular, tc seek solutions for his own teaching problems by utilizing those^ of the various 
theoretical approaches suggested which 7 warrant such use. Wholesale rejection of them would 
hardly be constructive; it would be preferable to propose better ones, ^ 



s document are 



J.-J.G. 
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^Jould you like to find out how much you know ^BEFORE/ studying the rest of the Handbook? 
One wajr of doing this is to try to answer the questions in a PRE-TEST .^ 

x * 

To do. so, read straight away the quescions on pages 167 to 170, 269 to A-, co 346 and 

479. to 482 and enter your' answers on the answer sheet, page 10. s 

If your score is low in this pre- test , this can be a source of satisfaction, since it may 

t 1 9 

show that you were right to start perusing this Handbook and that it will be worth your while 
continuing! 

You may also rest assured that your score in the post- test (page *613) will be another 
occasion for rejoicing, for^it will show an appreciable gain! 

If your score c in the pre-test is high, you should choose areas wtiich are still "uncertain 11 
(those for which you did not find the correct answer) and go into the matter more thoroughly. 



•This pre-test deals only with theoretical kno^leclge related to (p. 2) objectives, 
group 2. You will have an opportunity of gauging how far you have achieved the group 1 
objectives by doing other exercises which appear throughout the Handbook (blue pages). 
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ioZ EDUCATIONAL OBJECTIVES 



This initial chapter deals with the need for a clear definition of educatipr^objectiyes. < 
It emphasizes the fact that, as has been shown during experimental research in education, the 
most effective way of defining an educational objective is to describe the observable activity^ 
enabling the teacher and the learner to knew if, how and when the objective has been attained. 
It describes a classification of intellectual processes which c*n be of great help in arranging 
educational objectives. Those interested in this approach ..iould read the following works by; 

/ '* ■■ ' '-*' 

R. F. MAGER: / £V *- 

Preparing instructional objectives (1962) " ' 

2. . Goal analysis (1972) * 

3. Measuring instructional- intent (1973} (Chapter III, pages 15 to 46) 

Fearon Publishers ^California, U.S.A. 

4. ■ Criteria for the evaluation oX objectives in the educational of health personnel 

WHO. Technical Report Series. 1977, No. 608 

* * * 

After having studied this chapter and the reference documents mentioned you should be 

able to: ? 

define the following terms: educational objective; prerequisite level; 
institutional objective; specific objective* cognitive, affective, and psychomotor 
domain; 

a ' ' 

list the qualities and sources of an educational obtfective; 

& - dafine the main tasks of the members of the health t^m; * v 

a" ^identify errors in a list of educational objectives drawn up by a colleague; 



(jb define educational objectives at the, institutional level for tjhe establishment in 
which you are teaching; x 



Q classify educational objectives according to whether they bejLong primarily to the 

\ / 

affective, psychomotor, or cognitive domain; v j 

define specific objectives corresponding to a subject with /which you are familiar-. 

/ 
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If you are 
NOT CERTAIN 

of WHERE you are 





you may very 



well end up. 



SOMEWHERE ELSE 

(and not even know it) 
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Definition of educational objectives - an essential step 
before choosing teaching methods and a system of evaluation 



It seems reasonable to believe that an educational programme has more chance of being 

effective if its purposes have been clearly expressed. Experimental research in the field of 

evaluation indicates that it is not possible to measure the results obtained from an educational 
system if its objectives have not been explicitly defined 



Let us stress again that an educational programme must be shaped selectively in terms of 
the aims to be achieved. Each time the goal is modified, the programme too must be modified 
accordingly. 

Educational objectives ir.ust be generated on the basis of data obtained from different and 
multiple sources: health needs and resources of society, the scientific method and the progress 
of science, the students, the sociocul tural system, etc. 



poss 



Educational objectives must proceed from a prospective study of requir 
ibilities.l N 



emehts and 



The definition of these objectives is a subject which the universities in regard to the 
teaching of health sciences, havealways avoided tackling, not because they cpnsidered.it 
unimportant but because, apparently, they thought it went without saying. 1 * > 

The institutional objectives of a faculty of medicine, for example , # .are/ thgy say, 
axiomatic: "We train doctors of international quality. IX is notf necessary to develop the 
description any further; medicine is universal. " However, when we try to get the teachers to 
define a little more fully what they are talking about we see how wide and fundamental the 
divergencies are as soon as we leave the sphere of generalities. The conflicts between 
fundajnen talis ts and clinicians, between advocates of preventive and of curative medicine, are 
the result of those divergencies. This conflict becomes acute during the periodic curriculum 
reforms. 

The method traditionally used is to bring together eminent professors and the result of 
their deliberations is presented as a list of chapter headings. Often existing programmes are 
used as the main source of data for the preparation of Che, new programme. The professors 
indicate the number of hours to be devoted to the various subjects to be dealt with: this 
generally leads to a conflict of personalities and it is the most forceful, the most persuasive, 
sometimes the most irascible or noisiest of the participants in the. discussions who obtains the 
largest number of hours. 

The result is that the time variable becomes a cons lant and that competence becomes 
a variable. ^ ' 



Specialists in education, as a result inter alia of the studies which have stimulated 
progress ih cybernetics, have recognized the need' to establish models and in that same spirit 
a£ "new 1 ' technique is, proposed. 



\ 



1 

The epidemiological, sociological data, etc/, and the operational research necessary for 
such a prospective analysis are not dealt with in this HANDBOOK. For information in these . 
fields, the reader should consult specialized publications dealing with the organization of 
health services (see references p. 615 et seq «)« 
2 

For example, the length of medical studies is fixed by administrative regulations at 
6 years (or 7, 5 etc.): this is the time constant. On the other hand, there is hardly any 
definition of the competence of graduates, and this can lead to great variability. 
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106 EDUCATIONAL OBJECTIVES - 

Educational goals must be defined by using behavioural terms corresponding to the tasks 

to^be accomplished. In other words, the definition must indicate what the graduates of 

a given school will be able to DO at the end of their period of education or training thai they 
were not able to do before. 

These educational objectives are also called LEARNING objectives to distinguish them from j 

teaching objectives. They define what the student should be able to do, not the teacher. / 

They have been recently called COMPETENCY objectives referring to the functions required for 
the performance of a profession. 

A teacher's expectation of high student learning motivation will not become a reality 
until students feel a freedom to learn, that is to say when they become involved in the 
decisions concerning their own educational objectives. IJoreover, these objectives have to be 
explicitly defined ai.d offered for their choice; even better, the students \W\ have helped 
in establishing them. ' 

Educational objectives might be divided in different ways: 
* • 
If they describe the broad overall aspects of a medical school or any other centre 
of health sciences, they are called INSTITUTIONAL objectives. 1 ? -\ ^ I 

If, on the contrary, /they aref specif ic , at the level of a short learning period for' 
example, they are called INSTRUCTIONAL (or SPECIFIC) objectives. 

* - \ , *" 

In-between are INTERMEDIATE objectives, still broad, developed from institutional 

objectives at the level, for' example, of departments or divisions. ■ ~~~ " 

Some educational objectives are called ENABLING objectives: it describes 
a prerequisite level in relation to another objective. 

\ * 

^ey can also be divided according to the domain of the intellectual process to which they 
belong: 

affective domain = attitudes 
^ psychomotor domain = skills 
^ cognitive domain = knowledge 

These classifications have no other aim than to facilitate an analysis of the learning, 
process and help teachers to make educational choices. f 

It is true that human behaviour can rarely be neatly divided in terms of cognition and 
affect. This division is somewhat artificial in that neither teachers nor those responsible 
for programmes want to separate entirely these entitles. 

^ These differences will remain ^artificial so long as we do not possess the necessary 

* instruments to study both cognitive and affective outcomes simultaneously. It is to be hoped 



that an increased' emphasis on the definition of educational objectives will enable 

education researchers to resolve some of these problems and consequently assist in the choice of 

suitable teaching methods. * 

* / 

When educational objectives are, described in a practical and detailed manner it is then 

possible to determine with some precision those learning activities suitable for attaining an 

objective and -those which are almost or completely unsuitable/. 

! -> 
/ * 
The lack of an explicit definition of educational objectives makes discussions on 

programmes, teaching methods and evaluation methods completely invalid and futile. 



* There may be several for one institution: as many as the number of professional 
categories this institution is concerned with. 

bj4 23 ' 
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^ The methods of Revaluation (of students, teachers, and methods) will in fact depend on the 
objectives to be achieved. Evaluation consists in being able to say fo what MEASURABLE EXTENT 
and HOW the objective laid down has been achieved. If care has not bee.i taken to establish 
a MEASURABLE objective it will be difficult to make any kind of evaluation. To select a type 
of examination without educational objectives makes no more sense than trying to choose 
a measuring instrument without knoving what has to be measured. It is only during the last 
decades that investigators in the field of testing and measurement (docimology) began to work 
out a solution to the problem posed. Here, too, the strength of tradition and emotional 
reactions are making ^themselves fe^t. 

The following definitions, as well as the examples and exercises, are proposed as 
a basis for reflection and as an opportunity for practice in making a systematic approach to 
the solution of your educational problems. 



If the jargon seems obscure you should 
consult the glossary (page 601 et seq .) 
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Education is a PROCESS, the chief goal 



of which is to bring about change in 



human behaviour 



THE RESULT OF EDUCATIONS IS: 



AN EXPECTED CHANGE IN THE BEHAVIOUR OF THE 
STUDENT IN THE COURSE OF A GIVEN PERIOD 



f\ This behaviour will be explicitly defined in the form of educational 

Lyr OBJECTIVES : thev describe the expected performance. 

r\ A PROGRAMME of teaching/learning activities will be prepared and 

ly implemented to facilitate the attainment of these objectives. 

' 'A criterion reference EVALUATION system will make it possible to 

r\ measure the extent to which the- objectives have been attained - it 

ty will measure fi nal behaviour - and will also enable better educational 

' decisions to betaken . 
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Selection of goals 



There are essentiall> two ways of making this selection: one consists in relying on the 
judgement of experts to determine what a neophyte in the profession ought to know and ought to. 
be able to do. In the past we have relied almost exclusively on this method for determining 
the goals of medical education. As a reault curric . are crammed with an ever burgeoning 
quantity of new and highly specialized knowledge which the student perceives as irrelevant to 
his own goals and which, in fact, may be of little value to other than* the super sub-specialist * 
Certainl/ expert opinion is an important source of information about the knowledge and skills 
which trainees should be able to ^demonstrate , but it is also possible to make this decision on 
the basis of scientific evidence about what competent physicians need to know and need to be 
able to do in order to deliver responsible patient care. A number of procedures have now been 
developed for collecting such data to provide an empirical basis for developing a behavioural 
description. 0/ the essential components of professional competence to guide a faculty in 
setting their goals and designing their curricula. Three of these are of special interest: 
the critical incident technique, the method of task analysis and the analysis of epidemiological 
data. * 

The critical incident technique 

This method consists in collecting comprehensive data about specific- behaviours that 
characterize professional effectiveness and ineffectiveness and using these data to make an 
objective, empirical determination of the essential performance requirements of the profession. 
This^technique is an outgrowth of studies in aviation psychology made in the United States 
during World War II. In that programme it was found that in reporting the reasons for 
eliminating a trainee, pilot instructors and check pilots frequently offered such cliches and 
stereotypes as "lack of inherent, flying ability", "poor judgement" or "unsuitable temperament." 
In an effort to determine the specific qualifications of personnel that contributed to success 
or failure, combat veterans were asked to report incidents observed by them that involved 
behaviour which was especially helpful or especially ina equate in accomplishing the assigned 
mission. This request concluded with the statement: "Dtacribe the officer's action. . What 
did he do?" The several thousand incidents submitted in response to this inquiry were 
analyzed and categorized to provide a relatively objective and concrete description of the 
"critical requirements" of combat leadership. 

In applying this method to the medical profession, several thousand incidents describing 
observations of especially effective or especially ineffective colleague behaviour are 
collected from several hundred physicians representing various age ^groups, geographic areas, 
types of affiliations and specialty interests. For example, in a critical incident study of 
intern and resident performance .(i.e. of the general undifferentiated physician) commissioned 
by the U.S. National Board of Medical Examiners, the American Institute of Research who 
conducted the study collected over 3000 incidents from physicians across the country. The 
incidents submitted involved all areas of behaviour: cognitive, affective and psychomotor. 
They identified, for example, such general requisites of competence as "Skill in gathering 
clinical information," i.e.*, in taking a competent history and in performing an adequate 
physical examination, or "Skill in relating to the patient and in gaining his cooperation in 
a plan of manageir. 1." In an analogous study conducted by the University of Illinois Center 
for Educational Development of the critical performance requirements in orthopaedic surgery, 



From "An overview of appiied research in medical education problems, principles and 
priorities". Christine H. McCuire. WHO Report on the Workshop on the Needs for Research in 
Medical Education. Alexandria, March 1974. 
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over 1700 incidents were collected from over 1000 orthopaedic surgeons representing various 
practice settings and sub-specialty interests. An empirical, classification defining 
94 critical performance requirements, grouped into 9 major categories of competence, was / 
derived from the incidents. This operational definition of the essential components of , , 
competence could then be employed to determine the goals of specialty training, the design of 
programmes for their achievement and the criteria and methodology for their evaluation. If 
educational planning were regularly directed by such operationally defined, empirically 
derived goals, educational programmes would look substantially different., 

task analysis ^ * 

A second method of determining the essential components of professional competence whicfi*~ 
should define the goals of medical education consists in detailed task analysis of what 
physicians in various practice settings actually do, and in deriving from that statement of^ 
tasks a statement of the requisite knowledge and skills which they must haye in order to * 
perform these tasks competently. Such a task analysis may be based on careful, systematic 
observations of a representative sample of physicians in different practice settings* or it / L^ J 
be based on diary studies from the daily" logs of a representative sample of physicians who/ 
report in minute detail the way in which .they spend their professional days over a specified . 
period of time, or on some combination -of* .these two approaches. * 

Wherever this method has been employed, the results have been most enlightening. ' For 
example, in a limited pilot study of paediatricians in a typical small U.S. city, researchers 
found that all the physicians had different but consistent patterns for taking a history and * 
performing a physical examination. Of the 481 patient visits observed, 222 were well 
children; an average of 10.2 minutes were spent with these children (range: 7.5 minutes to ^ 
13*6 minutes) in contrast with an average of 8.1 minutes spent with ill children (range: . » v 
7.4 minutes to lo minutes). Of the 259 ill children, 104 (i.e. 40%) were diagnosed as having - 
an infection of the upper respiratory tract, 15 had chronic illnesses and 5 had potentially ^ 
life threatening diseases. For the total group of 481, optic fundi were examined- only ^tiroes 
and rectals were performed in only 6 cases ; 2 physicians did not percuss the lung fields for 
any patient. The greatest amount of time was spent in discussion of nutrition and child 
development. The single most frequent topic on which advice was rendered in well-child care 
concerned toilet training. The authors of this study concluded, "Few aspects of well-child 
cafe appear to require the skill of a physician . . . the question is also raised as to 
whether current training programmes are aggravating the physician manpower shortage by 
overtraining in relation to community health needs . "1 ~ 

This is a question that could be reiterated in every specialty in every country; only 
task analysis or comparable empirical sources will give us the answer. 

Epidemiological studies - 4 

One of the most interesting of the newer approaches to the use of such sources in 
determining goals of medical education consists in combining three arbitrarily weighted 
factors - disease incidence, individual disability and social disruption to define priorities 
in health care needs and, hence, in educational effort. As initially developed by 
Dr John.W. Williamsor^ the three factors are computed as follows: disease incidence consists 



* ?ergman, A., Probstfield, J. and Wedgewood, R. Performance analysis, in pediatric 
practice: preliminary report. Journal of Medical Education , Vol. 42: 262 (1967). 

Williamson', J. et al. Journal of American Medical Association , Vol.' 201: 938 (1967) 
and Vol. 204: 303 (1968). % , a 
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of a simple tabulation of the frequency of the disease (e.g. pneumonia) or other medical , 
condition (e.g. pregnancy) in the target population. Individual disability involves 
a determination of the extent of a patient impairment or risk associated with a given medical 
condition; an Individual Disability Weight (IDW) is calculated for each condition, from three 
elements: the .average length of hospital stay, mortality rates and complication rates. 
Social disruption represent^ an estimate, of the impairment thatjWOu,ld be produced by a given 
disease precondition in the larger social gr<5up of which the patient is a member; it is based 
on such factors as cost of illness, age of patient and number of dependents, socioeconomic 
standing and the like. For each discharged patient a Total Priority Weight (TPW) is 
calculated combining these elements. This Total Priority Weight is then arbitrarily 
apportioned among patient diagnoses. Finally, a cumulative total for each diagnosis is . 
, calculated from the' total patient sample. The resultant ranking represents a quantitative ^ 
estimate of priority or health care needs for the population at risk. 

It is clear that even with unlimited resources not all of these needs can be met in the 
present state of our knowledge. The next step therefore consists in defining the area of 
total health care needs which can be met, given our present* understanding of disease "and our 
present resources for therapeutic interventions. This area therefore defines the target area 
for application of professional understandings and skills and helps to define educational 
priorities. The goals of medical education for the basic undifferentiated physician can 
therefore be defined as encompassing those areas of health care needs which cause the greatest 
total preventable disability - i.e., those which cause the greatest total disruption that could 
be reduced or minimized by early diagnosis^and appropriate intervention. * / 

* * 

In his early studies using this methodology to review hospital practice in two large >> 
community hospitals in widely separated metropolitan areas in the United States, Dr Williamson 
found that pregnancy / involving uncomplicated delivery, ranked first or second in priority in 
both hospitals, th^t cerebral vascular accidents ranked ^mong the first five diagnostic 
categories in both hospitals and that fractures of the lower extremity ranked among the first „ 
five" in one hospital. These particular conditions are mentioned becauag, in some educational 
institutions there is a general tendency to reduce the amount of clinical instruction for the 
general medical student in some of these areas. For example, instruction in orthopaedic * 
surgery is often an elective despite the fact that trauma in general accounts for a very 
significant proportion of the total preventable disability. 

While the study reported above was limited to hospital practice, the same methodology 
l* could readil^ be applied to any level of health practice. Secondly, while the findings from 
such epidemiological studies and the particular weights to be assigned such factors as 
individual disability and social disruption will, of course, vary markedly in different parts 
of the world, the approach is clearly applicable to any population for whom health manpower are 
being trained. . But, in all parts of the world, utilization of such data will modify the goals 
and priorities of educational institutions and the emphases in medical curricula, by focusing 
far greater attention on ambulatory medicine and on more common causes of disability than are 
frequently seen in teaching hospitals. ^ 

J 

Implications of applied research on goals and priorities 

In the foregoing brief descriptions of methodology, it is suggested that the means are now 
at hajnd for supplementing expert judgement with data derived from empirical studies to assist 
us in defining the roles and, hence, the requisite competencies of the graduate^of pur 
programmes. Should such studies be generally undertaken and should their findings be optimally 
Employed in developing explicit goals and object!. _s of education in the health" prof essions , we 
would see revolutionary changes in the kinds of health professionals produced, and in their 
education programmes. Furthermore, such changes would have far greater impact in meeting the 
healt)i care needs of the populations than the simple expansion of educational facilities of the 
conventional type. 

Those interested more deeply in this subject could refer to Health Project Management, 
WHO Offset Publication No. 12, Geneva, 1974. * 
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ORGANIZATIONAL DIAGRAM SHOWING THE RELATIONSHIPS 
BETVjffcEN.JHE SUBSYSTEM "EDUCATION" 

(inside the broken-line quadrilateral) 
AND THE SUBSYSTEM "HEALTH SERVICE" 
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PREREQUISITE 



+7 • 



T 



WHAT THE LEARNER HAS TO BE ABLE TO 
"DO"* BEFORE UNDERTAKING AN t 

V. 

EDUCATIONAL PROGRAMME. 



* "TO DO", i.e. a group of behaviours from which it may be concluded 
1 that the learner has the requisite knowledge, skills and 

attitudes. f~ 
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I 'educational objectives 




W^AT THE STUDENTS SHOULD^BE ABLE TO DO 
AT THE END OF A LEARNING PERIOD THAT 
THEY COULD NOT DO BEFOREHAND 




The definition of the objective of a course is that of the result sought, 
not a description or summary of the programme. 



EDUCATIONAL OBJECTIVES 




Note: The size of the circles (left) is directly related to the amount (number) 
of objectives: the more they are specific the more numerous they are: 
the triangle (right) indicates that at INSTITUTIONAL level objectives 
are "wide", vague and that specific objectives are "punctual", narrow,, 
precise. - - ' 
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TYPES OF EDUCATIONAL OBJECTIVES 



1 



Institutional: at the level of the educational establishment (broad, 
all-inclusive). 



Example: 

Providing preventive and curative care to the individual and the community, 
in health and in sickness 



2 



'Intermediate": still broad, developed from the institutional objectives. 



Example: 

Planning and carrying out a blood sampling session for a group of adults 
in the community. 



3 



Specific (or instructional): corresponding to a given learning activity: 
precise and measurable. 



Example: 

? Using the syringe,to take a blood sample (5m i) from the cubital vein of an 
adult (criteria: absence of haematoma: amount of blood taken within 10 % 
of the amount required: not more than two attempts). 



IN ALL CASES THEY DESCRIBE 

V 

THE BEHAVIOUR OF THfE STUDENT 
NOT THE TEACHER 'S 
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DATA NECESSARY TO GENERATE 
EDUCATIONAL OBJECTIVES 



Health needs, demands and resources of society. 



Services to the patient. 



Service to' the community. 



The profession itself. 



The students. 



Progress in sciences. 



'The scientific method. 



etc. 
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INSTITUTIONAL OBJECTIVES 



The next seven pages give examples of INSTITUTIONAL educational objectives, of different 
sources* 

They are only examples. If they apply to the establishment where you are teaching it is 
by mere coincidence... or because the health needs of the populations for whom you are training 
.health manpower have much in common with those of the populations with whom the authors of the 
objectives* given as examples are concerned. J 

At the present stage this is unimportant; what should be noted is that all the examples 
are relatively short (one page) and relatively vague (they are also called general objectives) 
but that they all define what the .STUDENTS should be able to do at the end of their training. 
They do not define what the TEACHERS do but rather what the institution's "end product" is. 




Q EXERCISE ifr) BEFORE reading these institutional objectives take the time to list on page 121 
the f ivelaajor tasks of the physician (or nurse, or sanitary engineer ox village health 
assistant, according to the type of training in which you are interested). If there is a 
national "health plan, use it as a reference. If not, draw on your experience of your 
profession. 

After this exercise, read the examples of institutional objectives which follow . k They 
may seem vague and indefinite to you; indeed almost meaningless. Do not forget, however, 
' that they cannot be considered separately but form a WHOLE together with objectives at other 
levels (intermediate and in particular, specific). It is this constant linkage which gives 
them a meaning. Moreover, even allowing for their indefiniteness you will realize their 
' usefulness, on merely comparing them with what was taught (and still is taugjit) in many teaching 
I institutions. Indeed, how many of these objectives are seriously taken into account in the 
curricula of your school? 



WELL CONSTRUCTED INSTITUTIONAL OBJECTIVES ARE THE FOUNDATION 
OF A RELEVANT PROGRAMME 
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INSTITUTIONAL OBJECTIVES 



At the end of his M.D. Programme, the graduate will have acquired or developed the 
knowledge, abilities and attitudes necessary to qualify for further education in any medical 
career. He will also be able' to: 



1. Identify and define health, problenfs, and search for information to 
resolve or manage these problems* 

2. Given a health problem, to examine the underlying physical or behavio- 
ral mechanisms. A spectrum of phenomena might be included, from molecular 
events, to those involving the patient's family and community. 

3. Recognize, maintain, and develop personal characteristics and attitudes 
required tor professional life. These include: 

- V 

(a) C awareness of; personal assets, potential, limitations and 
emotional reactions. 

w 

(b) responsibility and dependability. » 1 

f " 

(c) ability to relate to and show concern* for other individuals. 

A } * j 

4. Develop the clinical skills and methods required to define and manage- 
the health problems of patients, Including their.'physical , emotionaL, and 
social aspects. 

5. Be a self-directed learner, recognizing personal educational needs, 
selecting appropriate learning resources, and evaluating progress. 

6. Critically assess professional activity related to patient care, 
health care delivery, and medical research. 

/ 

7. Function as a productive member of a small group, which is engaged 
in learning, research or health care. 

*• * 

8. Work in a variety of health' care settings. 



From McMaster University, Canada, June 1972. 
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125 



Upon completion of the^ formal course o£ studies, a graduate of the M. D. programme will 
have acquired or developed knowledge, abilities and attitudes so that he will be able to ; 



A. 


Perform professional services within a community health care system compatible 
with the overall policies of the Center -for Health Sciences: 




1. 


Identify ;and define present and future community health problems and work 
to resolve such problems by the planning, implementation ani evaluation 
of preventive or remedial programmes. 




2. 


Use clinical skills, knowledge, original observations and appropriate 
records to identify, diagnose, manage (prevent, refer or treat ratio- 
nally) and follow-up the health problems of his patients, taking into 
account the physical, psychological and sociocultural aspects. 




3. ■' 


Work as a leading partner in a health care team. 




4. 


Educate the population and motivate them to improve their health. 


B. 


Continually increase fris level of competence: / 




1. 


Take part in postgraduate training (residency programmes, specially 
zation, courses, etc.) and reaching (students and colleagues). 


- 


2; 


Periodically evaluate his professional activities, recognize his " / 
educational needs, select appropriate learning resources and evaluate 
his progress. ^ 


C. 


Aid the development of the health sciences by engagingin teaching and 
research and seek solutions to new health problems of his patients, 
community or health care system with which he is not familiar. 


D. 

« 


Maintain and develop personal characteristics and attitudes required for 
professional life, such as personal integrity, sense of responsibility and 
dependability, and ability to relate to, communicate with and show concern 
and respect t>o his patients and colleagues. 



1 Ben Gurion University of the Negev, University Center for Health Sciences, Beer-sheba, 
Isra'el. 
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INSTITUTIONAL OBJECTIVES 1 
the end of their Doctor in Pharmacy programme the graduates should be able to : 



1. Monitor drug therapy, including the prevention, detection, and correction 
of drug-related problems. 

2. Manage and optimize drug therapy relative\o the total needs of the r ^ 
patient, in collaboration with the physician, nurse, and other health 
professionals, r— 

3. Provide supportive clinical services such as drug information, drug 
surveillance, and ^rug delivery and distribution. 

4. Develop the administrative systems to maximise the use of physical, 
fiscal, and human resources to implement pharmaceutical services. 

5. Assume delegated responsibilities for the drug therapy management \of 
patients with specific diseases. This involves screening, evaluation, 
managing, and follow-up procedures in diseases such as hypertension, diabetes 
mellitus, arthritis, and certain mental illnesses. 

6. Set therapeutic objectives, and monitor patients for desired therapeutic 
response; design, and utilize acceptable drug therapy management 
protocols. 

7. Collect, analyse, interpret, and evaluate appropriate subjective and 
objective data necessary to identify the patient's therapeutic needs; 
obtain patient drug histories and apply these histories along with those 
taken by others, drug record systems, and other patient data related to 
drug therapy management. 

8. Evaluate the effect of drug therapy on major disease processes; evaluate 
how the signs and symptoms may be modified by drug therapy. 

9. Detect drug-induced disease processes and differentiate these from other 
disease processes. ^ 

10. Detect and evaluate the clinical importance of adverse drug reactions 
and interactions; provide effective and appropriate management solutions 
to these problems and contribute to their prevention. . 

11. Analyse the epidemiological aspects of adverse drug reactions and 
evaluate drug usage patterns; develop and use drug surveillance reporting 
systems for the detection, management, and prevention of adverse drug 
reactions wjlth particular emphasis on the objective methods and systems 

to monitor potential and incident adverse effects. 

12. Use, and critically evaluate current drug studies in the medical and 
pharmaceutical literature. 



They should be able to perform these functions in all types of practices. 



From the Doctor of Pharmacy Program Proposal submitted to the State Council of Higher 
Education by The School of Pharmacy, Medical College of Virginia, Virginia Commonwealth 
University, ,19 August 1974. 
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INSTITUTIONAL OBJECTIVES 1 

r * 



At the end of her School of Nursing Baccalaureate Program the graduate should be able to : 



i. 

i. ' 


Provide effective nursing care to people of $11 ages at all, levels of 
wellness in a variety of settings. 

Apply the nursing process in the delivery of health care to individuals 
and groups.. 


» 


3, 


Demonstrate an effective use of self £n her interactions with others. 




_ 4. 


Collaborate with individuals and groups in reaching realistic health 
goals.,, 




5. 


Initiate change to improve the quality of professional nursing practice 
and the delivery of health care. * ' 




6. 


Seek opportunities for continual growth as a person, a citizen, and a 
professional. 

> • * 




7. 


Practise nursing within a framework which is congruent with her own 
philosophy of nursing. 





t 



/ 



■ j 

r 



From Boston University Bulletin 1974/75. 
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INSTITUTIONAL OBJECTIVES \ h>\ 

At the end of her Undergraduate^ school of nursing programme, the nurse should be able to: ^ 



1. Assess with individuals and groups, their health-illness status and context in 
order to determine nursing care implications. ' 

2. Collaborate with others to synthesize plans, to improve health care. 

3. Formulate a plan of nursing .care which contributes to the total plan* of health 
care. * . 

4. Implement plans fpr health and nursing care within broad health care plajis or*;* 



systems. 



* 5. \ Implement teaching to improve nursing and^nealth care. ^ 

<f 6. Evaluate the effectiveness of nursingc^are and health plans and systems. 

7. Develop and maintain helpful relationships with individuals that would facilitate 
health care. t • ' 

'8* Use research knowledge applicable to nursing and health care. % 

9. Apply research skills to solve and/or study nursing and lealth problems. 

10. Appreciate the historical aspects of the profession of nursing and health care 
and their relationship to current and futuristic goals in the delivery of 
health care service. - , , 

11. toe appropriate independent leadership and collaborative role relationships as 
^indicated by the goa,ls to be accomplished. 

12. Demonstrate concern for the uniqueness and rights of individuals and groups in 
relation to health care. 



13. Develop continually self-awareness. 

1^. Continue developing the ability to learn and being responsible for own learning. 

i5.. Use social actions with responsibility to bring about changes in the interest 
s of promoting health. * t 

16." Use dynamic technological advances to improve nursing and health care t . 



1 Adapted from the University of Washington Bulletin, 1973/75. 
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Description of Practitioner of Tomorrow 



The graduate of the new baccalaureate nursing programme will be prepared to 
function as a generallst with beginning competencies in a specialized area of 
nursing* 

The graduate will be prepared to function in a variety of settings and be 
able to; 



1. Obtain health histories and make general health assessments, 

2. Provide safe and competent care in emergency situations and acute illnesses. 

3. Provide supportive care to persons with chronic or terminal health problems. 

4. Provide health teaching, guidance and counselling. 
5.. Assist persons to maintain ^optimal health status. 

6. Provide for continuity of health services. 

7. Assume leadership. responsibility for planning and evaluating nursing care. 
Work effectively with all persons concerned with health care pr/fblems. 



8 



This baccalaureate nurse, as a practitioner of nursing, will be accountable and 
responsible to clients for the quality of nursing whether administered directly or 
indirectly. 



/ 



* From the University of Washington School of Nursing. 
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INSTITUTIONAL OBJECTIVES 
At the end of his training the Public Health Dental Assistant 1 should be able to : 



1. Educational - organize and provide to individuals and groups oral health 
education, adapted to age, cultural, and socioeconomic factors, on oral hygiene 
practices, preventive oral measures, nutrition, accident prevention, consequences 
of ritual dental mutilations, oral health as a part of total health. 

2. Diagnostic ^ 

2.1 differentiate between oral conditions which the FHDA is able to treat and 
those requiring referral 

2.2 diagnose periodontal diseases (gingivitis, periodontitis and periodontal 

abscesses) \ 

i 

2.3 diagnose dental caries and resulting conditions (pulpitis, apical periodo- 
ntitis, and acute apical abscesses) \ f 

2.4 make preliminary diagnosis of traumatic injuries to structures of the oral 
cavity \ 

2.5 take, process and interpret radiographs of teeth and surrounding structures. 

S 

3. Preventive and curative 
3.1 scale and polish teeth 

3*2 apply topical preventive agents 

3.3 administer local anaesthetics, analgesics and specified medicaments for 
treatment of oral diseases 

3.4 perform simple extractions of teeth and treat specified complications 

3.5 perform simple cavity preparations and fillings 

3.6 perform intra-oral incision and drainage of dental abscesses, place and 
remove sutures 

3.7 provide emergency dental treatment within the scope of training and initial 
treatment of traumatic injuries 

3.8 perform cardiopulmonary resuscitation according to local instructions. 

4. Administrative 

4.1 chart diagnosed conditions and record a patient treatment plan indicating 
current and all completed dental treatment 

4.2 .clean and sterilize dental instruments and equipment 

4.3 maintain dental supplies (instruments, expendable supplies, medicaments, 
surp^ry equipment) 

4.4 perrorm simple repairs of dental equipment 

4.5 submit periodic summaries for monitoring the prevalence and^other changes in 
disease experience 

4.6 prepare and submit through channels periodic reports on the services performed 
by. the FHDA 

4.7 complete administrative procedures for patients requiring referral 
dental care. 



1 School for Public Health Dental Assistants, Kampala, Uganda. 
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T 



draw up INSTITUTIONAL educational (student oriented) objectives suitable 
far^.the establishment where you are^ teaching and based on the FIVE main 
tasks you have described (page 121). 



r 
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INTERMEDIATE OBJECTIVES 

,1 



OBJECTIVE OF THE DEPARTMENT OF EPIDEMIOLOGY AND P REVENTIVE MEDICINE 

: : ■ / 

At the end of his university studies the student should have tjie motivation for and 
be able to : 

, / 
— » 'plan and undertake, in cooperation with other members of the health team 

/ < ' 

epidemiological studies of an analytic or descriptive nature; 

> evaluate the results of tljese studies within the framework of the economic and 

/ 
/ 

social structure of the regional tfr nationa community and describe them in ecological terms; 

plan and undertake, with the aid of other members of the heal.h team, the 

application of preventive measures against the communicable ar.d nutritional diseases as well 

/ 

/ 

as against the noncommuni Cable diseases (occupational or hereditary diseases), according to 

i 

their epidemiological priority; 

■ H » improve health (health promotion), i.e. improve the wellbeing of the population as 

a whole, even in the absence, pathologically speaking, of any specific disease. 



/ 



-2* 



University Health Sciences Centre (CUSS), Yaounde, United Republic of Cameroon. 
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* INTERMEDIATE OBJECTIVES 1 
(BASIC NEUROSURGERY) 



A graduate 'should be able' to : 



1 # Perform and record a neurological examination, and decide whether the data 
elicited support or confirm the presence of a disorder in the central or 
peripheral nervous system. 

2. Suspect from patient data the presence of urgent' remedial neurosurgical 
situations 

(a) Acute and chronic intracranial haematoma 

(b) Hydrocephalus 

(c) Mass effects producing brain or spinal cord distortions 

*3. Select the appropriate clinical and laboratory procedures essential to the 
clinical evaluation of the suspected neurosurgical disorders. 

4. Describe the hazards, attributes, and limitations of the laboratory procedures 
commonly employed in the clinical investigation of neurosurgical disorders (EEG, 
echoencephalography., lumbar puncture, pneumoencephalography, ventriculography, 
arteriography,' myelography and radiography of skull and spine, and isotope 
scanning) * 

o 

5. Perform a lumbar puncture demonstrating proper techniques and proper recording 
of pressures. 

6. Perform cell count, Pandy test, and evaluate bloody spinal fluid. 

7. Initiate a satisfactory course or treatment for injuries of the brain, spinal 
■ cord, and its coverings. 

8. Describe pathology Of the common neurosurgical conditions. 

9. Describe the principles of prevention, recognition, and treatment of the 
metabolic, cardiovascular, pulmonary, gastrointestinal, cutaneous, locomotor, 
and genito-urinary disorders associated with central nervous system failure, 

'lO. Extrapolate from the symptoms of pain its informational (diagnostic) value. 

11. Demonstrate a knowledge of the reasonable clinical expectation to be derived 
from operative neurological surgery. 



University of Southern California Medical School, Los Angeles. 
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INTERMEDIATE OBJECTIVES 1 
(MEDICAL PHYSIOLOGY) 



, Experimental methods and principles 

1. Given Green's (an*introduct£on to human physiology), Ganong's (a review of human 
physiology) and notes supplied during the course, define all woras and concepts mentioned 
in the text. 

2. State a principal experimental approach by means of which knowledge concerning the function 
, of any orgari'-cbiild have been acquired. 

3. Presented wi'th *a set of relevant data be able to: 



^a) 



put up an"*equation based upon the law of energy or mass conservation allowing a 
determination of the blood flow of the organ in question, 

(b) describe in words what the equation states, A 

(c) decide which assumptions must be made for the equation to be valid, ^ 

(d) solve the equation, 

fe) calculate blood flow values in both "static" and "dynamic" "systems". 

i 4 

\ 

Presented with data on physiologically relevant problems or concepts contained in the 
syllabus 

(a) give a systematical description of such data shown in figures, graphs, and tables. 

(b) decide, which conclusions - particularly regarding cause/effect relationships - tha£. 
may be drawn solely on the basis of the ^data presented, i.e. without regard to the usual 
physiological concepts and theories. 



(c) ^ecide which conclusions may be drawn with respect to whether the observations are 
consistent with the accepted theories, i.e. whether the observations may "be explained" 
by means of the usual textbook conceptions of the relationships between the phenomena 
observed. Or instead, give reasons for his/her own conceptions at variance with those 
~^6f the textbooks. 




Re. 4. on dSrawartion of data: 

with access to a textbook of statistics : 

4.'1 comparfe two sets of results giving the level of significance of the difference 
between them and give the presumptions for the statistical comparison, { 

4.2 given an article, paper, or similar dealing with elements or parameters of the 
syllabus be able to make a summary, adhering to a presented set of standard criteria 
supplied by a number of scientific associations. 

5. Concerning clinically relevant applications 

Presented with a description of a case the student should be able to: 

1 

b.l point out symptoms and signs signalling abnormal function of the body or of its 
organs, 



Institute of Physiology, University of Aarhus, DK 8000 C^Aarhus, Denmark. 
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• * 

5.2 (a) explain how to collect pertinent #ditional information with the purpose of . 
deciding whether abnormal function is present and eventually clear up, 

(b) explain which functional elements/partial functions are abnormal, 

(c) give the principal causes of such abnormal function, 

(d) indicate where the affection may be localized, 

(e) distinguish between subjective and objective types of examinations/ data, 

5.3 presented with data from such examinations 

(a) interpret the data, i.e. explain what each parameter may be an expression of 
and give the principal causes of the deviations present, 

(b) formulate a pato-physiologic "diagnosis" (hypothesis) that explains the data 
presented* 

5.4 by setting up a programme of examinations contained in the syllabus given, list 
them according to priority beginning with the simplest and ending up with the most * 
complicated (e.g. when estimating renal function measuring urine volume and osmolarity 
before making any direct clearance determinations. 

f 

Clinically relevant facts (probably not contained in the syllabus) 

6.1 state, that regular use is necessary with any functional element in the body to 
maintain optimal function and give examples of this from both afferent and efferent CNS 
functions, « 

6.2 define subjective clinical methods as methods that include the cooperation of the 
patient (e.g. stating of sensory impressions), % 

6.3 define objective clinical methods as opposite of 6.2. 

Regarding experimental lab course s. 

Presented with material of the same type as produced or seen by the practical classes, 
the student should be able to: 

7.1 state the principle of methods employed, e.g. the P C 02- electrode is ba8ed on 
measuring the pH of a weak bicarbonate solution the P C o 2 ° f which is in equilibrium with 
th&t of the measured sample, 
* 7,2 define specificity, sensitivity, accuracy and precision as stated below, 
7.3 given information of how results have been obtained explain 

(a) whether t?* results show signs of systematic errors, 

(b) if yes, point out in which part of the procedure the error may be located 
(experimental or analytic error most likely). 
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SPECIFIC OBJECTIVES 



What are the qualities of a specific educational OBJECTIVE 




An instructional objective MUST have ALL the following qualities: 

RELEVANT : It should be based on the educational needs of the learner and 
the institutional objectives. This means stripping away all 
unnecessary material and making sure that useful points are 
" covered. 

LOGICAL : The objective must be internally consistent. 

UNEQUIVOCAL : Its meaning must be clear to the reader, and as there are many 
"loaded" words (uords open to a wide range of interpretation) 
when we use such words we leave ourselves open to misinterpretation. 

What do we mean when we say we want a learner to "know" something? 
Do we want him to be able to recite, or to solve, or to construct? 
Just to -tell him we want him to "know" tells him little or can 
mean many things. Until you describe what the\learner will be 
DOING when demonstrating that he "knows" or "understands" you have 
described very little. * * . 

FEASIBLE : It is essential to ascertain that what the student is required to 
do can ACTUALLY be done. It should be achiev«le within the time 
available and with the facilities at hand. Remember also the 
prerequisites of feasibility: the minimum level a learner 
should possess to ~qu alif v for the course (attitudes, skills and 
knowledge). This is the prerequisite level ., 

OBSERVABLE : It is, in fact, obvious that if the objective cannot be observed 
by some means it will not' be possible to ascertain whether it has 
been achieved. 

C 

MEASURABLE : One very often hears "many of the things I teach are intangible 
and CANNOT BE MEASURED". Even gross measurement is better than 
no measurement at all, for if no measurement is made, instructors 
tend to assume that a goal is being achieved because they have 
taught. If your teaching- skills cannot be evaluated, you are* in ^ 
the awkward position of being unable to demonstrate that you are 
4 teaching anything at all. That is why there is a need for 

including in the objective an indication of minimally acceptable 
performance. 
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QUALITIES OF A SPECIFIC EDUCATIONAL OBJECTIVE 



9 

:RLC 



Relevant 




Logical 



Unequivocal 



Feasible 



Observable 
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WORDS OFT|EN USED BUT OPEN 
TO MANY INTERPRETATIONS 



TO KNOW 



4 



TO UNDERSTAND 



- TO REALLY UNDERSTAND 



TO APPRECIATE 



- TO FULLY APPRECIATE 



i 



TO BELIEVE 



TO HAVE FAITH IN 



ETC ... 



ERIC 
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WORDS OPEN TO FEWER INTERPRETATIONS 



- TO WRITE 



- TO RECITE 



- TO IDENTIFY 



- TO DIFFERENTIATE 

- TO SOLVE 

- TO CONSTRUCT 



- TO LIST 



TO COMPARE 



TO CONTRACT 
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LIST OF ACTIVE VERBS FOR STATING EDUCATIONAL OBJECTIVES 





- 


nonrestrictive list - 




Abbreviate 


'Derive 


Insert 


~ Record 


Act 


Describe 


Integrate 


Reconstruct 4 


Administer 


Design 


Isolate 


Recount 


Aid 


Designate 


Justify 


Reduce 


Allow for 


Detect 


Label 


Regroup 


Analyse 


Determine 


Lead 


Relate 


Apply 


Develop 


List ~~ 


Remove 1 


Appraise 


Diagnose 


Locate 


* Reorder 


Arrange 


Diagram 


Maintain 


Reorganize 


•Assist— — - — 


— — .Differentiate 


_ .Make_ . 


Rephrase 


Ask 


Direct ' > 


Manipulate 


Replace 


Assemble 


Discover 


Map 


Request 


Attend 


Discriminate 


Match 


Reset 


Audit; 


Display 


Measure 


Resolve 


Avoid 


Dissect 


Meet 


Respond 


Bring 


Distinguish 


Mobilize 


Restate 


Build . 


^Divide 


Modify 


Safeguard 


Calculate . 


Do 


Move 


Select 


Care for 


Draw 


Name 


Send 


Categorize 


Edit 


Note 


Separate 


Change 


Effect • 


Narrate 


Serve 7 


Chart 


Encourage 


Obtain 


Set * 


Check 


EnumeraJ>€^^ 


* Omit 


Share 


Choose 


Enunciate 


Operate 


Simplify 


Circle 


Establish 


Oppose 


Solve 


Cite - 


Estimate 


Order 


Sort 


Clean 


Evaluate 


Organize 


Speak 


Close 


Examine ( 


Outline 


Specify 


Collaborate 


Exchange 


Paraphrase 


Start 


Collect 


Execute 


Participate 


State \ 


Communicate 


Explain 


Perform 


Store 


Compare • 


Extract 


Pick 


Structure 


Compile * 


Extrapolate 


Place 


Suggest 


Complete 


r Facilitate 


Plan 


Supply 


Compute 


Fill 


Play 


Support 


Conclude 


Find 


P-loy 


Synthesize 


Conduct 


Follow 


Point 


Tabulate 


Connect 


Formulate 


* Position 


Take responsibility 


Construct 


Furnish 


Practise 


Teach 


Contrast 


Generalize 


Predict 


Tie 


Contribute 


Generate 


Prepare 


Time 


Control 


Get 


Present 


Trace 


Convert 


Give • - 


Prevent 


Translate 


Cooperate 


Guide 


Prognose 


Treat 


Correct 


• Hold 


Promote 


Underline 


Create 


Identify 


Protect 


Use 


Criticize 


Illustrate 


Provide 


Utilize 


Decide 


Implant 


Pursue 


Verify 


Decrease 


Include 


- Put 


Wash 


Deduce 


Increase 


Raise 


Weigh 


Defend 


Indicate 


Read 


Work 


Define 


Induce 


Tftecite 


• •Write 


Delimit 


Infer 


Reassure 




Demonstrate 


' Inform 


Rearrange 





i 



i 
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RELEVANCE 



IS THE ESSENTIAL QUALITY 



OF EDUCATIONAL OBJECTIVES 




OBJECTIVES WHICH HAVE. EVERY 



57 



EDUCATIONAL 



TO SAY WHAT ONE MEANS BY A GOAL 

\ 



IS NEITHER TO REDUCE THE IMPORTANCE 



OF THE GOAL OR ITS PROFUNDITY. 



THE ACT OF WRITING IT DOWN MEANS MERELY 

V 



THAT WHAT WAS ONCE SECRET IS NOW OPEN 



FOR INSPECTION AND IMPROVEMENT 



(MAGER, 1972) 
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„ ELEMENTS OF A LEARNING 
OBJECTIVE 



ACTIVITY 



* - CONTENT 



CONDITION 



4fc - CRITERIA 



0 



9 
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f DESCRIPTION "OF THE ACTIVITY* 

It is perfectly acceptable for the ACTIVITY (of a specific objective) to be stated as a 
" performance indicator 11 and^not the actual activity provided that the latter is perfectly 
evident. 

- The actual activity is the description of the task aimed at by a given objective: 

Exaiuple^ = "repair a binocular microscope' 1 

*" Tne Performance indicator is the description of an act from observation of which the 
ability to perform the actual activity can be inferred: 

Example = "underline in the diagram the names of all veins that pass IN FRONT of the 
corresponding artery" 

Here it is quite obvious that if the activity consists in "underlining" it is only a 
performance indicator from which can be inferred whether the student is able "to distinguish 
veins possing IN FRONT from veins in some other position". 

In all cases, the procedure to be followed as concerns the ACTIVITY (actual or performance 
indicator) is as below: 

1. Identify the activity (for example, by underlining it in the sentence). 

2. Decide whether it is an actual activity or a performance indicator , 

2.1 If it is a performance indicator, decide whether it enables an inference to 
be made concerning the actual activity, 

(a) If so, decide whether it can be simplified and whether it corresponds 
well to the student's level, 

(b) If not, write another one. 

c 

2.2 If it is an actual activity, decide whether it is explicit or implicit, 

(a) If it is explicit, decide whether it can be simplified and whether it 
corresponds well to the student's level, 

(b) If it is implicit, include a "performance indicator". 




For more details see MAGER, measuring instructional intent , pp, 24-38, 
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Let us take an example of a specific educational objective and identify each of its elements : 

"Identify, from memory, on a drawing of a horizontal section of the neck passing through 
the seventh cervical vertebra the name of all the vessels (arterial, venous and 
■ lymphatic) as shown £h Professor X f s book/' 



*ACTIVITY = 
* CONTENT = 
* CONDITION = 

* CRITERION = 



"Identify the name of"... 

Cervical anatomy, level of the C7 vertebra. 



Definition of the main conditions under which the act should take^place (data, 
restrictions and limitations); in this particular case the student must do 
the identification "from MEMORY". 

Definition of the MINIMUM LEVEL OF PERFORMANCE of the student. In this 
particular case he must identify ALL the. vessels, the reference being a given 
author. 



^Sxercise^ i 



identify each element of the following objective: 



to be able to: repair a binocular microscope (brand X, Y or 2) having been informed of 
the defect and given a descriptive diagram, appropriate tools and spare parts, so that 
the microscope functions according to specifications. 



The following pages include examples of speciiic objectives as prepared by 
workshops participants or found in documents produced by medical schools. 
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The student should 



be able to: 



1. List, without referring to a textbook, at least three important symptoms of pre-eclampsia. 

2. Reae^ one issue of a professiopal journal eveiy fortnight, outside working hours, and write 
a summary Xnot more than' 10 lines) of at least one of the articles. 

3. Give oral care with the available equipment to a patient confined to bed and conscious, 
according to technical sheet X. 

4. Find out the health needs of a family in its environment, by paying six domiciliary vxsits. 

(a) Write out in chronological' order all the stages of the domiciliary technique. 

(b) Apply the technique described to the previous item and prepare a plan of action. 

57^ Make a survey of the habits and customs of a population so as to identify those which 
"represent a health danger. < 

(a) . Prepare questionnaires. 

(b) Carry out the survey. 

(c) Interpret the results. 

6« During a prenatal consultation, carry out examinations for detecting and preventing 
complications of pregnancy (technical sheet Y) . 

7. ^Interpret the clinical signs of an extra-uterine pregnancy described in the report on a 
patient and make a differential diagnosis on the basis of the clinical picture accompanying 
the report. 

8. Give two simiJar and two dissimilar characteristics concerning: 

(a) the immunological value of 

(b) the epidemiological indications for 

inactivated, and 
attenuated, polio vaccines. 

9. Measure the height when lying down of a newborn baby, with a maximum error of 1%. 

10. Construct, using simple, already tabulated data, a HISTOGRAM including title, coordinates, 
additional details and a correct, neat and readable curve* 

* 

11. Identify by microscopic examination three of the following micro-organisms: meningococcus, 
Hansen's bacillus, human tubercle bacillus, and gonococcus. 

12. Using an optical microscope (magnification x 40) diagnose granulation tissue on a sLide 
in five minutes, indicating at least five points of recognition present on the slide. 

13. Identify in the course of a dissection all the vessels and nerves in the axilla which it 
is absolutely essential not to injure. 

14. Make a macroscopic diagnosis o r a benign breast tu»aour in an operation specimen and 
indicate at least four characteristics of benignity observed. 
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TAXONOMY OF INTELLUCTUAL PROCESSES 1 



1 



1. DEFINITION AND VALUE OF TAXONOMIC CLASSIFICATIONS 

A taxonomy is a bferarchical 'classification in a given field. Taxonomies in the field 
->f education provide a classification of various instructional objectives, at suitable levels, 
in^jfiven spheres. This constitutes a descriptive system based on logical principles in 
^accordance with the existing data in education and psychology* 

To help teachers formulate precisely the educational objectives, taxonomies of learning 
levels have been developed by various groups concerned with educational problems. 

One of their reasons was to try and clear up the confusion which resulted from the use of 
ambiguous propositions such as "the student should have a good understanding (or knowledge) of 
the principles of 

"Good understanding of the law of X" may signify for some that the student should be 
capable of giving the mathematical formula for the law or saying what it means, and for others 
that he should also be able to use the formula to solve a problem or show the interdependence 
of the law in relation to other phenomena. No one questions the nee& for students to have a 
"good understanding" of a particular phenomenon, law or principle, fact or theory. However, 
what satisfies one teacher and enables him to say whether 4 a student has in fact achieved a 
"good understanding" may be yery different from what would satisfy another teacher, unless 
they have agreed on what they mean by "good understanding" through the use of a non-ambiguous 
terminology. 

Education has often been criticized for the imprecision of its terminology and concepts. 
The chief value of a taxonomy is that it enables educators to communicate among each other 
more precisely. 

This better understanding will make possible the collective work of those responsible 
for curriculum construction. A taxonomy could also be of use in evaluating the results of a 
system of education, through classification of educational objectives given on the basis of a 
standard taxonomy. Definition and evaluation of the quality of the instruction in a school, 
country or region poses a very difficult problem; and a well-designed taxonomy can play an 
important role in finding a solution. Another important function of a taxonomy is to help 
educators to construct examinations for students who have received instruction. A taxonomy 
is also useful in general as a research tool in education and evaluation. 



Summary of an article by Dr J. -J. Guilbert (AFRO internal publication, WHO, 1970, 
18 pages - in French; DIDAKTA MEDICA, February 1971 - in English, Formazione Permanente del 
Medico, April 1974 - in Italian). 
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2. THE DIFFERENT TAXONOMIES OF EDUCATIONAL OBJECTIVES 

2.1 Cognitive domain ^ 

The firt* category in this classification is knowledge . Knowledge involves the recall 
of specifics and universals, and of methods and processes, or the remembering of a pattern, 
structure or setting. 

Objectives that emphasize the memorization of facts, principles, processes and patterns 
in different subject areas belong to this category' of the taxonomy. 

The second category is comprehension * It iric^jles grasping communication accurately, 
putting it into a different form of presentation, reorganizing material in summary form without 
changing the central meaning of the communication, and extrapolation* 

The third category concerns the application or use of general ideas, principles or' methods 
to new situations. 

The fourth category concerns analysis * This consists essentially of the breakdown of 
material into its constituent parts, and detection of the relationships of the parts and of the 
way they are organized. ; 

The fifth category is synthesis , i.e. the ability to put parts or elements together to 
form a coherent whole. 

The last category is that of evaluation . It is defined as "the making of judgements 
about the value, for some purpose, of ideas, works, solutions, methods, material, etc* It 
involves the use of criteria as well as standards for appraising the extent to which particulars 
are accurate ... or satisfying. The judgements may be either quantitative or qualitative, 
and the criteria may be either those determined by the student or those which are given to 
him M . 2 



4 



1 Bloom, B. S. (ed.), Taxonomy of Educational Objectives, Handbook I: The Cognitive 
Domain, Drvid McKa?, New York, 1956. 
2 

Q Ibid., Bloom, page 185. Qrf 
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COGNITIVE DOMAIN 



1. Knowledge 

1.1 Knowledge of specifics 

1.2 Knowledge of ways and means of dealing with specifics 

1.3 Knowledge of universals and abstractions or symbols in a 
field . 



2. Comprehension 

2.1 Translation* 

2 . 2 Int erpr etat ion 

♦ 

2.3 Extrapolation 
3* Application 

4. Analysis 

4.1 Analysis of elements 

4*2 Analysis of relationships 

4.3 Analysis of organizational principles 

5. Synthesis f . 

5.1 Production of a unique communication 

5.2 Production of a plan, or proposed set of operations 

5.3 Derivation of a set of abstract relations 

6. Evaluation 

6.1 Judgement in terms of internal evidence 

6.2 Judgement in terms of external criteria 

(going beyond the field of knowledge of the discipline and 
involving the ^rultural norms of science as a whole) 
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2.J..2 In 1963, C. McGuire proposed a simpler taxonomy derived from that of Bloom and more 
appropriate for the preparation of achievement tests for medical students. 

Level 1. . Knowledge 



1.1 Recall 

# 

1.2 Recognition of meaning 

2. Generalization 

3. Problem solving of a familiar type 

3.1 Interpretation of data 

3.2 . Application 

4. Problem solving of an unfamiliar type 

4.1 Analysis of data 
4«£ Special application 

5. Evaluation 

6. Synthesis 

In 1967, the same author further simplified the classification, proposing four levels: 
1 # Recall of facts, or generalization 

2. Interpretation of specific data 

3. Problem solving 

4. Evaluation or synthesis of a total situation 




Three) levels seem to satisfy the- needs of evaluation 



Centre for Education Development, University of Illinois Medical School. 



TAXONOMIC LEVEL 

COGNITIVE DOMAIN 



1. RECALL OF FACTS 



2. INTERPRETATION OF DATA 



3. PROBLEM SOLVING 



i 
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2.2 Affective domain 1 

This taxonomy comprises five main categories. ^ 

The first category, that of "receiving " or attending , includes sensitivity to the 
existence of a certain phenomenon or stimulus, and awareness, which is almost a cognitive 
behaviour. r .t also includes willingness to receive, or attention. 

The second category is responding . At this level the learner is sufficiently involved 
in a "subject, phenomenon or activity that he will seek it out and gain satisfaction from 
working with it or engaging in it" . ^ ' 
/ 

The third # category is valuing . At this level the behaviour is consistent and stable, 

and it involves not only the acceptance of a value but also preference for it and a commitment 

to or conviction for a certain point of view or ideal. 

• * 

The fourth category, organization , concerns the level at which the learner constructs a 
value system whicfi guides his behaviour. 

The fifth category is characterization by a value or value complex . "At this level of 
internalization the values already have a place in the individual's value Hierarchy, are 
organized into some kind of internally consistent system, have controlled the behaviour of the 
individual for a sufficient time that he has adapted to behaving this way; and an evocation 
,of the (value system and related) behaviour no longer arouses emotion or affect, except when 
the individual is threatened or challenged".-* 

< A simplified, three- level classification is proposed, restricted to: 

1. receiving 

2. responding 

3. internalization. 



1 [%* Jiwohl, D. R. . 3loom, ;i. S., Masia, 15. B. , Taxonomy of Educational Objectives. 
iic'riV'CK T : effective Dona in, David MacKay Company, Inc., New York, 1964. 

2 V iJ., P. 173. 

3 # 

Ibid., p. iHL. 
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AFFECTIVE DOMAIN 



1. 


Receiving (Attending) 


• 




1.1 Awareness 

1.2 Willingness to receive 

1.3 Controlled or selected .attention 




~2. 


Responding 

O 1 An/mi' oc^onro "in rpQnnnni no 
L . I nLQUlcoLciiLc ill Lcopuiiuiug 

2.2 Willingness to respond 

2.3 Satisfaction in response 




3, 


Valuing 

3.1 Acceptance of a value 

3.2 Preference for a value 

3.3 Commitment (conviction) 




4. 


Organization 

4.1 Conceptualization of a value 

4.2 Organization of a value system 




5. 


Characterization by a value or value complex 

5.1 Generalized set 

5.2 Characterisation 
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TAXONOMIC LEVEL 



AFFECTIVE DOMAIN 



1. RECEIVING 



2. RESPONDING 



INTERNALIZATION 
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2.3 Psychomotor domain 1 

The proposed classification of psychomotor behaviour comprises five categories."' 

The first category is imitation . When the learner is exposed to an observable action, 
-he, begins to make a covert imitation of the action at the level of his muscular system, 
guided by_ an impulse to imitate it. 

The second category is that of manipulation . At this stage, the learner is able to 
perform an act according to instruction and not merely on the basis of observation. He also 
begins to differentiate between one set of acts and another, becomes capable of selecting the 
fcctrrequired, and begins to attain skill in manipulating selected implements. 

■5 

In the third category, called precision , the proficiency of performance reaches a higher 
level of refinement in reproducing a given act. 

The fourth category concerns articulation . It involves, essentially, the coordination 
of a series of acts by the establishment of an appropriate sequence (internal consistency) 
among different acts. 

The fifth category concerns naturalization of a single act or a series of articulated 
acts. The skill of performance attains its highest of proficiency, and the act is performed 
with minimum expenditure of energy. 

A simplified, three-level classification is suggested , t limited to: 

1. imitation 

2. control 

3. automatism. 



1 .R. H. Dave, International Conference of Educational Testing, Berlin, 16-24.5.67. 
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PSYCHOMOTOR DOMAIN 




1 

1 > 


Lull. LuLlUU 

1.1 Impulsion 

1.2 Overt repetition 










2. 


Manipulation 

2.1 Following direction 

2.2 * Selection t 
* 2.3 Fixation ' 










3. 


Precision 

3.1 Reproduction 

3.2 Control 


1 




6 




4. 


Articulation 

4.1 Sequence 

4.2 Harmony 








. V 


5. 


Naturalization 

5.1 Automatism 

5.2 Inter iorizat ion 
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EDUCATIONAL OBJECTIVES 



TAXONOMIC LEVEL 



PSYCHOMOTOR DOMAIN. 



1. IMITATION 



CONTROL 



3. AUTOMATISM 



EDUCATIONAL OBJECTIVES 



TRIPARTITE STRUCTURE 

The complete plan of classification applied to educational objectives 
comprises three taxonomies, represented by the following diagram: 



TRIPARTITE TAXONOMIC SCHEME 
OF EDUCATIONAL OBJECTIVES 



EVALUATION 

1 



SYNTHESIS 



ANALYSIS 



APPLICATION 



COMPREHENSION 



KNOWLEDGE 



T 



CHARACTERIZATION 



ORGANIZATION 



VALUING 



RESPONDING 



RECEIVING 



NATURALIZATION 






ARTICULATION 






PRECISION 


i 




MANIPULATION 


> 




IMITATION 



EXPOSITION TO AN 
EDUCATIONAL EXPERIENCE 



7^ 



EDUCATIONAL OBJECTIVES 



PROBLEM SOLVING 







INTERPRETATION 
OF DATA 






RECALL OF * 
FACTS 



COGNITIVE 
DOMAIN 




THE 3 TAXONOMIES 
(SIMPLIFIED) 



INTERNALIZATION 







RESPONSE 






RLCE 


PTION 



AFFECTIVE 
DOMAIN 



AUTOMATISM 







CONTROL 






IMIT/ 


WON 




PSYCHOMOTOR 
DOMAIN 

















EXPOSITION TO AN 
EDUCATIONAL EXPERIENCE 






NOTE: Please remember that in hMman behaviour the 3 domains are 
intricately connected. 



EDUCATIONAL objectives 




/ 



/ 

/ 



IF YOU WANT MORE DETAILS ON MOW TO FORMULATE 
OH TO ASSESS EDUCATIONAL OBJECTIVES REFER 




// 




Criteria for the evaluation of learning objectives in the education of health 
personnel. Report of a WHO Study Group. World Health Organization Technical 
Report Series , 1977, No. 608 (ISBN 92 A 120608X) . U pages. Price: Sw.fr. 6.-, 
US$ 2.40. French, Russian and Spanish editions ii. preparation. 



How can a medical school, or indeed any school that trains health workers, ensure that 
the education it offers will be suited to the needs and demands of the population to be 
served by its graduates? One well-known but still insufficiently used tool for ensuring 
this kind of relevance is the learning objective, also called the educational obj^jc>We. 
To take the definition used in a recently published WHO Study Group^r^porJ^^r^Tearning 
objective is a "statement describing the expected resultsoJ^^rrtTIn^e^eriences is they 
manifest themselves in student performance or bejja-v4<nrr^ The, Study^Group in question was 
convened by WHO to discuss the uses of learning objectives in the education of .hea 1th personnel 
of all kinds and, more specifically, to draw up guidelines for the evaluation of such objectives. 

The guidelines devised by the Study Croup inquire about not merely the wording of 
educational objectives but mainly the manner in which they were developed/ Were the 
objectives developed so as to reflect, for example, the expectations of health care workers 
and consumers? Does each objective use an action verb that clearly describes what the 
student must do to show that he has achieved the oDjective? Is the set of objectives as a whole 
internally consistent? The guidelines consist of 37 such items. For many items, the usei 
can simply check off one of the printed answers. A space for "remarks" is included under 



An annex to the report, based on a working paper prepared by one of the Group's 
temporary advisers, clarifies what different authors mean by learning objectives, examines 
the different levels and types of objectives, lists the potential benefits of taking the 
trouble to formulate objectives, and reviews the data considered necessary in order to 
derive objectives. There is also a short section on how to word learning objectives properly. 
The annex concludes with a et of recommendations that were used by the Study Group in 
developing its guidelines. ^ 

The report and particularly the guidelines should De of value to administrators and 
teachers in schools for health personnel who wish to draw up or evaluate educt lon'.l 
objectives for their own use and that of their students. 



each item. 
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Now 




Define nine specific educational objectives concerning a subject with which you are 
familiar, stating explicitly what you feel the student should be able "to do" at the 
end of a given course (that he was not able to do previously) and corresponding to the 
cognitive, affective , and psychomotor domains; make sure, in particular, that they 
include all the necessary elements ( activity , content , condition , criteria ) 1 at three 
different levels. 



1st level 

- cognitive domain 

- psychomotor domain 

- affective domain 
2nd level 

- cognitive domain 

- psychomotor domain 

- affective domain 
3rd level 

- cognitive domain 

- psychomotor domain 

- affective domain 



1 Review if necessary the first chapter (pp. 138-148) concerning the quality cf a 
specific learning objective. 



7- 
♦ / 
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EDUCATIONAL OBJECTIVES 



» CHECK YOUR RESULTS 

Do you feel that each of the nine specific objectives satisfies the following conditions: 
Hois it correspond to the domain chosen (affective, cognitive, psychomotor), include the 

four elements : activity, content, condition and criteria, and is it student orientated?. If 

so, put a cross in the corresponding box. 



OBJECTIVE 


HOMA I N 

\J\JMi\ X 

& 

LEVEL 


STUDENT 
ORIENTED 


ACTIVITY 


CONTENT 


CONDITION 


CRITERIA 


1 














2 














3 














4 














5 














6 














7 














8 














9 















You can also make the same analysis concerning the specific objectives cited on page 149- 
This "table" is also called a SPECIFICATION TABLE - for more details see page 255. 
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EXERCISIy (check your answers on page 172) 




Question 2. 





Instructions: for each question select one answer. 



An educational objective corresponds to all the follov/in^ elements EXCEPT ONE . 
Indicate which: 

A. It is sometimes also called a learning objective* » 

B. " It defines explicitly what the teacher should do. 

C. It should be the basis for the preparation of a curriculum. 

D. It can be institutional or specific. 

E. It is defined in behavioural terms corresponding to the tasks to be 
accomplished. 

Mager stresses the need to define educational objectives in operational terms 
(description of the behaviours aimed at). Among the following qualities select 
the one which does not correspond to oae ox the advantages sought by this 
technique. 

A. It enables the student to evaluate his progress throughout his period 
of learning. 

B. It enables the instructor to choose relevant examination questions. 

C. It enables the instructor to make a satisfactory choice of teaching 
methods, 

D. It enables the student to acquire a thorough knowledge of the content, 
organization and timetable of a course. 

E. It enables the student to show clearly, at the end of the course, the 
acquirement or lack of specific behaviours. 

A specific educational objective has all the following qualities EXCEPT ONE . 
Indicate which: 

A. It clearly communicates an educational intent. 

B. It makes objective evaluation possible. 

C. It defines a measurable behaviour. 

D. It establishes success criteria. 

E. It facilitates analysis of functions and tasks. 

Indicate which of the following statements does not correspond to the principles 
of rational and effective educational planning; 

A. Institutional objectives should be defined before selecting a student 
evaluation system. 

Institutional objectives should be derived from the objectives of the 
departments or chairs of a school. 



B. 
C. 
D. 
E. 



Regional health needs should be considered before drawing up an 
educational system for training members of the health team. 

Educational objectives should be defined before selecting educational 
methods and learning activities. 

National resources and budgetary constraints should be taken into 
account before defining institutional objectives. 
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Question 5. 




Question 7. 



Indicate which of the following definitions corresponds to the prerequisite 
level: 

A. What the student should -be able to dc at the end of the curriculum. 

B. What the teacher should summarize before every course so as to place 
all the students on the same level. 

C. What level the student should attain in the process of a given 
educational programme. 

D. What the student should be able to do before undertaking an 
educational programme. 

E. None of the above. 

Which of the following statements best corresponds to the educational aim to be 
achieved: 

A. The student is perfectly familiar with the anatomical structure of 
the thorax. 

B. The student is aware of the socioeconomic factors associated with 
portal hypertension secondary to acquired liver disease. 

C. The student describes in writing the various steps of a nursing care 
plan in relation to the needs of a patient. 

D. The student has a thorough knowledge of the differences between 
a normal and a premature infant. 

E. The student has a positive attitude towards patients. 

Indicate which of the following statements best corresponds to £he fcjr 
qualities of an educational objective defined in behavioural terms: * 
\ 

A. The teacher will give five lectures of an hour each to a group of 
20 undergraduate students. These lectures will deal with the 
physiopatho logical mechanisms of burns. 

B. During a simulation exercise, the student will give a brief written 
definition of the problem (at least 25 words) and list at least three 
alternative solutions, indicating which he would adopt and giving the 
reasons for his choice. 

C. The student will show his knowledge of "the mechanism of the action of 
aspirin on the subcortical cells. 

D. The student will give a written list of. at least five of the seven 
branches of the radial nerve. 

E. The student will demonstrate to his teacher, at the patient's bedside, 
that he has a good grasp of clinical nethod and a sound critical 
judgement, without his relationship tD the patient suffering thereby. 



so 
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There have been numerous classifications of the intellectual process. One of them 
considers three domains: 

1. affective domain 

2. psychomotor domain 

3. cognitive domain 

Using the following code: 

A = 1 D = 1 and 3 * 

B = 2 E = 2 and 3 

C = 3 

indicate the domain(s) to which the following educational objectives correspond: 

Question 8. The student should be able to name four new ideas concerning tuberculosis control. 

Question 9. The student should be able to measure the head circumference of a newborn infant 
with a maximum error of 1 cm. , 

Question 10. The student should be able, when contraceptives are requested by an 18-year-old 
girl, to give information on the choice of available methods without expressing 
any moral judgement. 

Question 11. The student should be able to avoid giving a guilt feeling to a five-year-old 
enuretic child. 

Question 12. The student should be able to construct, using simple data already tabulated, 
, a histogram including title, coordinates and additional details without any 

mistake in* the curve. 

Question 13. The student should be able to prepare a health education poster, given a model 
and the necessary materials (paper, felt, charcoal, coloured pencils, glue, 
scissors). 



Question 14. 



Which of the following statements is in contradiction to the theories expounded 




The change in behaviour resulting from a learning activity is called 
performance. 

B. Final behaviour is the designation of the observable act which will be 
accepted as proof that the student has achieved a given educational 
objective. 

C. The definition of the objective of a course is a description or 
* summary of the programme. 

V 

D. The teaching intent explained by the educational objective indicates 
what should be the final behaviour of the student. 

E. None of the above statements. * 
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Instructions : 



The following group of questions consists of lettered headings followed by 
a list of numbered words or statements. For each numbered word or statement 
select the one heading that is most closely associated with it and blacken 
the corresponding space on the answer sheet. Each lettered heading *nay be 
selected once, more than once, or not at all. 



Indicate the equivalent terras: 

A. Educational objectives 

B. Institutional objectives 

C. Performance 

D. Specific objectives 

E # Minimum level of performance 

Question 15. Terminal behaviour. 

Question 16. Instructional objectives. 

Question 17. Competency objectives. 

Question 18. Criterion. 

Question 19. Learning objectives. 

Question 2C. Acceptable minimal level. 



★★★ 
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Answers suggested for the exercise 
on pages 167 to 170 



QUESTION 


ANSWER 
SUGGESTED 


If you have not found the correct 
answer, refer once more to the 
following pages 


1 
i. 


B 


105-106 


2 ' 


D 


106,109, 116-1*7' 


3 


E 


105-106-107, 114*; 138 


4 


B 


. 106-107 , 111 .120 




n 
u 


105,116,138 

•* 


6 


c 


138 to 142 


7 


D 


146 to 148 


8 


c 


\ 




q 


B 






10 


D 




105 to 107 * - 


11 


A 




f and 151 to 163 


12 


E 






13 


B 






14 


C 


108-109, 116-117 


15 


C - 


108,138,148 


16 


D 


106.108, 116-117 


17 


A 


106 


18 


E 


108-109, 148 


19 


A 


106 


20 


E 


138 
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At the cost of some effort you very likely have been able to attain most of the 
objectives indicated on page 102; this is the first step, and that is why it is very ^ ! 
important, but there is still a long way to go. Even to define all the specific objectives 
of your present teaching w^.11 be a long job. And after that you must make sure tha,t they 
correspond to institutional objectives, whether explicit or not ♦ . . and it would be more 
reassuring to know that these objectives are definitely related to the health problems of 
tomorrow's population. Never forget the relevance . This is not alii The most recent 
studies on the subject seem to show that teachers do not know how to UTILIZE the educational 
objectives they have defined. Thousands of specific objectives may be defined in vain if 
they do not subsequently serve as a reference for preparing the educational activities which 
will result in their being achieved. 

Other awkward questions are: do these objectives which you have defined actually 
correspond to what is really important or do" they merely reflect what is (relatively) easy to 
set out in the form of objectives? Will they really help the students and will the latter be 
better trained than if . . . etc. 

You will be faced with these arguments at some time or other, generally by those who do 
not make any attempt to define their own objectives or to learn how to use them. Whatever 
may be the limits and drawbacks of this approach it, has the irreplaceable advantage of enabling 
studies* to be made and research to be carried on. , In order to reply to the questions 
mentioned above, and many others, it will be necessary, and this is a matter of urgency, to 
carry on research whose scientific rigour will ij/crease the credibility of the arguments, 
theories and hypotheses advanced by , "education specialists". For no matter how logical 
arguments may be, they must be backed, by some proof. Consequently any attempt at dogmatism 
in this field or wish to define objectives at all costs, should be avoided. These extremely 
necessary studies will call for considerable experimentation. 

\ 

You can help in carrying out such experiments. 

If all this has not discouraged you, go on to the next chapter. Good lucki 
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'••D 0 NOT FORGET! 
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PLANNING OF TEACHING/LEARNING ACTIVITIES 
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PROGRAMME 



This second chapter deals with the planning of teaching programmes and learning 
activities. It attempts to bring out the change which is now taking place, i.e. the tendency 
to be interested more in what the student receives, perceives and assimilates than in what 
the teacher presents, gives or does. It chows that this change can be of benefit to all 
(the teacher continuing to learn and the student taking over part of the teacher's role)*- 
It deals with the teaching- learning process as part of a teaching-learning systenu It 

•a • 

stresses the effectiveness of methods which place the student in an ACTIVE situation. It 

f * ' 

proposes a collective plan of action for drawing up a curriculum taking into account the 

♦ 

(traditional) obstacles to change. 

Those with a deeper interest in these problems are strongly advised to consult the 
following publications: t J . 

Public uealth Papers No. 52, WHO, "Development of educational programmes for the 

IT 

health professions", 1973 

Public Health Papers No. 47, WHO, "Aspects of medical education in developing 
countries", 1972 * 

Public Health Papers No. 61, WHO, "Educational strategies for the health 
professions", 1974 

Technical Report Series No. 489, WHO, "Implications of individual and small group 
learning systems in medical education", 1972 

Technical Report Series No. 521, WHO, "Training and preparation of teachers in 
schools of medicine an/i of allied health sciences", 1973 
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After having studied this chapter and the references mentioned you should be aole to: 
Jl^ identify the differences between "education", "teaching", and "learning"; 

|2i describe the new trends of the ^teaching/learning system; 

cite 10 conditions which facilitate learning; 

|4i indicate the aims and general methods of teaching; 

identify at least two advantages and two limitations of certain techniques employed 
in teaching; * . * 

i 

select an appropriate teaching technique for facilitating the attainment of an 
educational objective; \ 

draw up an organizational diagram showing what you think can be done to introduce an 
"integrated teaching curriculum" into your establishment with consideration of 
constraints and^Vtva.tegies to overcome them; 



0 



6, 

identify which role, as a teacher, you intend to assume in order to facilitate the 
learning of students for which you are responsible; 

identify the obstacles liable to be encountered during the setting up of a 
competency based curriculum; 

complete a curriculum specification table. 
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THE PURPOSE OF TEACHING IS TO FACILITATE LEARN LNG L 



The Literature on the philosophy of education, is rich in theories, which tell a story of 
timid steps forward, backward leap* and rediscover ies. It would be very gratifying to have 
a reliable general theory, firmly sea Led on a scientific basis and making proper allowance for 
social variables, which could serve as a guide for ever> teacher and enable him to resolve the 
"real" problems of teaching the health professions* Unfortunately, such a universally 
satisfactory general theory does not exist. On the o*ther ha. by means of the systematic 
approach suggested, hypoLheses can be formulated regarding the process of acquiring 
a satisfactory level of performance that can be evaluated empirically and the choice of 
learning activities facilitated * 

The contemporary trend is to stress the ,f teaching- learning system" as opposed to the 
preponderance previously given to teaching alone. There is a tendancy to be interested less 
in teaching Than in learning, less in what t h e teacher presents and more in what the student 
learns . Lawrence M. Stolurow has criticized what he calls the "communication-learning 
fallacy" which assumes that the information transmitted to the student is a lways learned. 
Doubtless this is obviously fallacious, and known to be so when it is expressed so crudely, 
but discussions o c . teaching methods are often still inspired by it and it has even been 
carried over into the initial stages of research into new methods. Much of this research, by 
concentrating on problems of the presentation of stimulating materials and utilizing some of 
the more rudimentary concepts of communications theory, dealt too much with the manner in 
which information was transmitted to the student without investigating very closely wh^t ./as 
learned and by whom, at what speed and, in particular, for what purpose . Understandably, 
this approach led to a passive attitude towards students' response; the student was *een in 
a dependent situation, relying upon information directed at him, whether through modern 
audiovisual communication techniques or the more traditional forms of the lecture and the text 
book. 

Learning, however, is a dynamic arrd fn teractivc r iov«s in which the behaviour^and 
experience of the student are vital components; the student must -not only receive but also 
contribute; his perception of what is happening is just as important as the perception of his 
teachers and the assessment he T.akes of the val^e of a learning activity may be more relevant 
than that of his examiners . Good conventional teaching, of course, has always sought to take 
account of the learner, but its structure and methods have greatly inhibited it. The rigid 
style imposed by large numbers, timetable requirements and the availability of teaching space, 
by the conventional practice^ in designing courses and by teaching conforming to an accepted 
academic discipline, have led to the "teaching" aspect again dominating over the "learning" 
aspect. it is accepted that tl.e starting point must be the acquisition of knowledge rather 

than its communication then we i.iust ask so;r,e different questions and intensif> our investigation 
of NON-DIRECTIVE methods. Using the ^ame approach a Jerome S. fircner uc can consider what 
experiences will motivate the student and enable him to learn, in what ways knowledge can best 
be structured for a given student or ^roup of students, what sequence and in what form the 
material can be presented moot efK wt -w'ely, what should be the nature and the frequency 
of rewards or penalties, and W*„ w ear. gradually lead a student to give less thought to 
extrinsic rewards than to ' , cr^jaal satisfaction of having achieved a desired degree of 
skill. 

The body of knowledge possessed b^ a group of teachers 01* set out in a series of 
authoritative volumes io the fruit of intense intellectual activity. Teaching a so-called 
basic science is not a matter of getting the student to memorize it, but rather of helping him 
to participate in a process that renders the acquisition of a body o£ knowledge possible. 
A subject is taught not to produce little living libraries on that subject but rather to get 



Adapted from Norman Mackenzie, et al. Teaching and learning , Paris, UNESCO, 1970, 
pp. 44-50. 
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the student to think for himself in accordance with the laws of physics, to consider problems 
from the same angle as the biologist and to assimilate the process of acquiring knowledge. 
Knowing is a process, not a product* 

If the revision and renewal of the programme is regarded as part of the teaching- learning 
process, a change must also occur in the roles and interrelationships of teachers, students 
and others who are concerned system become^ much mure complex than the conventional 
linear model and it demands much more from those who have to design, manage and operate it* 
To put it epigrammaticaxly s the teacher becomes a learner himself, and the learner undertakes 
some part of the teaching role . This is because the teacher learns more about teaching and 
the student begins to assume a greater responsibility for his own progress. The roles of 
other participants also change and new roles must be added to the old. Some of the new 
methods at present under trial, for instance, not only require librarians to emerge from their 
traditional sphere into a more dynamic type of activity; they also call for the services of 
highly qualified personnel, such as artists, photographers and technicians, as well as 
educational technologists and psycho legists. In fact, the ecology of a teaching institution 
changes once its primary function is redefined , namely to facilitate the acquirement of 
competence by the student. 




/ 



/ 



In the following pages (207-216) a certain number of definitions, 
epig'rams - and lists are proposed whose aim is to serve, where 
appropriate, as a starting point for reflections or discussions 
on the methodology of education. These elementary concepts are 
derived from various systems, and none of them is regarded as 
embodying an absolute and final truth. 
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TEA C H I N G: 



*** INTERACTIONS 



between TEACHER and STUDENT 



in order to bring 



EXPECTED CHANGES IN BEHAVIOUR 



OF THE STUDENT 



0 
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PURPOSE OF TEACHING 



to |iIe^students T0[) 

A Q U I R E, RETAIN AND BE ABLE TO USE KNOWLEDGE 



UNDERSTAND, ANALYSE, SYNTHETIZE AND EVALUATE 



ACHIEVE SKILLS 



ESTABLISH HABITS 



DEVELOP ATTITUDES 

★ 
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TEACHING APPROACHES 
® TALK JTO STUDENTS 

® TALK WITH STUDENTS 

^ HAVE THEM TALK TOGETHER 

§1 SHOW STUDENTS HOW 

^ SUPERVISE THEM 

® PROVIDE OPPORTUNITIES FOR PRACTICE 

9i 
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★ 

. . . TEACHING METHODS 

WHICH PLACE THE STUDENT IN AN 

SITUATION FOR LEARNING 

ARE MORE LIKELY TO BE EFFECTIVE 
THAN THOSE WHICH DO NOT 

G.E.MILLER 
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^ LEARNING 



• • • 



A PROCESS 



resultiny in some MODIFICATION 



relatively permanent 



of the BEHAVIOUR, 



way of 



THINKING, 



of the LEARNER 



FEELING, 



DOING, 
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THE CHARACTERISTICS OF LEARNING ARE 
THAT LEARNING IS : 



1) Producing a behavioural change in the learner 



2) Leading to a relatively permanent change 



•3) Resulting from practice and repetitions 



4) Not directly observable 
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SOME PRINCIPLES OF LEARNING 



* 

1. Learning is individual 

/ 

/• 

I 

2. Motivation is the key 

/ ■ 



3. Relevance of learning experience 
should be clear to the student 



4. "Feedback" to learner is 



important 



93 
ERIC ,. 



.PROGRAMME • , 215 



CONDITIONS to facilitate 
LEARNING — ► 



• « « 



A N 



ATMOSPHERE 



WHICH 



* 



encourages people to be ACTIVE 
emphasized the PERSONAL nature of 'learning 
accepts that DIFFERENCE is desirable 
recognizes people'^ RIGHT to make mistakes 
tolerates IMPERFECTION 

encourages OPENNESS p SELF and TRUST in SELF 
makes people feel RESPECTED and ACCEPTED 
facilitates DISCOVERY i 

puts emphasis on SELF EVALUATION in COOPERATION 
permits CONFRONTATION 1 



G.P. Pine & P.J. Home, Principles and Conditions for. Learning 
in Adult Education - Adult Leadership. Oct. 69 
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LEARNING 



is primarily controlled by the learner 



* is unique 2nd individual 

* is affected by. the total state of the learner 

* is cooperative and collaborative 

* is an evolutionary process 

* is a consequence of experience/ 

* is not directly observable 



GJ. PINE & P.J. HORNE (ibid.) 
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* Exercise 



i 

1 - Complete in less than five lines the following sentence: 



"I learn best when I 



2 - Ask several colleagues to do the same exercise; put all the "situations"" on the' same 

list. I 

3 - Note the diversity of situ t:ions and compare it with that of che educational 

institution to which you belong. 
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NOTE CONCERNING THE PREPARATION OF "TRANSPARENCIES" 



Certain pages of the Handbook are printed in large type and can be used as matrices if 
you should wish to make your own "transparencies" for projection using an overhead projector. 
There are different methods, depending on the photocopying equipment available. 

In general, all you need do is to place a sheet of transparent, thermosensitive acetate 
film on top of the Handbook matrix 1 and put the whole in the photocopying machine. In a few 
seconds you will have a transparency ready to be projected on your overhead projector. 

Please follow, the instructions applying to the particular photocopying machine available 
to you. 



1 You can obtain them by removing the staples from the Handbook and placing the sheets 
in a ring-file. 
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TEACHING TECHNIQUES 



] 



The Handbook does not give detailed descriptions of the different teaching techniques 
employed in centres for training health personnel. This is intentional, for an abundant 
literature is available on these subjects to which the reader is referred (see references) • 
Another reason is that the main purpose, has been to stress the relevant aim of the teaching 
(whence the almost obsessional insistence on the need to define'educational objectives) 
(chapter I) and on the means for checking if and how these aims have been achieved 
(chapters III and IV), The problem is not merely one of COMMUNICATING better (this is the 
-subject of chapter II) but- of better communicating a RELEVANT MESSAGE* 



f ' 



As a teaching technique WORKSHOPS are certainly a very useful and efficient one. For 
details see chapter 5. 
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TEACHING TECHNIQUES 

For details concerning teaching techniques, lectures, seminars, tutorials, practicals, 
programme learning, etc., the reader should refer to the specialized publications (see 
references, page 515 et sea .). The following references are also suggested among the 
documentation cited on page 202. 
w The lecture method of instruction - Eileen Bughman 
Public Health Papers No. 52, WHO, pp. 57-63. 

W Audiovisual aids to learning - E. B. Penta & T. V. Telder 

Public Health Papers No. 52, WHO, pp. 40-51, 
9 The selection of teaching/learning materials in health sciences education. 

Technical Report Series No. 538, WHO, 
A Techniques to implement a course design model (Unit C of) "Systematic Course Design", 

A. SEGALL, John Wiley & Sons, 197 j. 
m Teaching and Training methods for management development 

IL0 Geneva Third impression, 1974. 
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ADVANTAGES AND DISADVANTAGES OF CERTAIN TEACHING METHODS 



\ \ 


ADVANTAGES 


DISADVANTAGES 


ir Lectures (1) 

*■ 


- Apparent saving of time (for the 
teacher) and resources 

- Presence of the teacher 
(showmanship) 

- Covers a large group of students 

- Gives feeling of security 

- Enables a large amount of information 
to be presented 

< 


- Keeps the student in 
a passive situation 

- Does not- facilitate 
learning problem- solving 

- Offers hardly any 

* possibility of checking 
learning progress 

- Does not allow for 
individual pace of learning 

- Low receptivity 


it Practical wortc 

it Bedside 

teaching (3) 
* Field work (4) 


- Puts the student in an active 
situation 

- Covers a limited group of students 

- Permits evaluation of degree to which 
educational objectives have been 
attained 

- Develops qualities of observation and 
decision-taking 

- Ensures closer contact with reality 
(professional, health situation of 
country, colleagues and teachers) 

- Permits comparison between reality 
and theory 

- Enables student to develop self- 
confidence 

- Increases variability 


- High personnel costs 

- Sometimes puts the patient 
in a difficult situation 

- Poor standardization 
-Narrow limits ci 

utilization 

- 


^ Small group 
activities 
(5) 


- Permits a teacher/student dialogue « 
(thanks to the availability of the 
teacher) 

- Facilitates evaluation 


- High costs in personnel 
and time 


it Programmed 
* learning (from 
book's) (6) and 
^ simulation (7) 


- Enables student to work at his own 
pace 

- High efficiency 

- Facilitates self- evaluation 
Permits mass teaching 

- nign avanaD.LJ.ity 

- Facilitates decision- taking (solution) 
of complex problems 

- Decreases risk (for patient or 
society) 


- Necessitates special 
educational competence 

- High additional investment 
costs (in teachers*' time 
and money) 

- No group dynamics 
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From the seven techniques indicated on page 221, choose the one (cr ones) which you feel 
appropriate for each of the 14 specific objectives mentioned on page 149 



Obj ective 


Technique 


Constraints to be overcome locally 


1 






2 






3 - 


• 




, 4 


i 




5 






6 






7 




• 


8 

\ 
\ 






9 




. *v 


10 






11 






12 • 






13 






14 







Also indicate the constraints which exist in your teaching situation and which are an 
obstacle to the application of the technique selected. 



ERIC 



106 



"PROGRAMME - 



ADVANTAGES AND LIMITATIONS OF DIFFERENT MEDIA 


For convenience of use, the list of media is divided into three main groups; 
non-projected, projected and sound. The "projected" group is further subdivided into still 
and moving pictures: the "sound 1 ' group refers to sound alone (most of the moving picture 
group incorporate sound). 


I. Non-projected media 


Media 


Advantages 


Limitations 


A. Books, 

handouts and 
other printed 
v matter 


1. Some learn best through reading. 

2. Allows seLf-pacing. 

3. Good for reference and revision. 

4. Handouts easily produced, duplicated, 
revised, for large number of students; 
can also be associated with teaching 
to reduce need for notes; can be 
reproduced in local languages. 


1. Published textbooks 
expensive, and sometimes > 
involve foreign currency 
problems. 

2. Published textbooks 
rapidly out of date and 
only revised rarely. 

3. Good manuals and 
handouts demand good 

f- \m i' no anH t"Pnt"rtHnCl n<* 

facilities* 


B. Real objects 
and specimens 


1. Present reality, not substitutes. 

2. Tnree dimensional. 

3. Permit use of all senses in study.— — 


1. May not be easily 
obtainable. 

2. Inconvenience cf s'ze - 
danger in use. 

3. Costly or not 
expendable. 

4. Usually only usable in 
f:.^all groups. 

S Sompt* i mp^ easilv 

damaged. 
6. Problems in storage. 


C. Models and 
simulation 
devices 

* 


1. Three dimensional and concept of 
reality. 

2. Size allows close examination. 

3. Good for magnified situation 
(e.g. middle ear mechanism). 

4. Can be used to demonstrate function 
as well as construction. 

5. Can permit learning and practice of 
different" technique, 

6. Some can be made with local material. 


1 # Craftsmanship required 
for local construction. 

2. < Simulation models often 
* expensive. 

3. Usable for small groups. 
4 # Models often easily 

damaged. 
5. Never same as performing 
technique on a patient. 
Beware wrong learning. 


D« Graphics 
(charts, 
diagrams , 
schematic 
drawings) 
posters , 
paintings, 
photographic 
prints 


1. Promote correlation of information. 

2. Assist organization of material. 

3. Photographs nearer to reali , than 
drawings, but association of^en 

^ valuable. 

4. Usually easily produced and duplicated 
(black and white photos). 

5. Easy to store, catalogue and retrieve. 


1. For small audiences only 
(unless projected with 
epidiascope). 

2. For effective use, good 
duplicating equipment 
and trained staff 
needed. 
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. Media 


Advantages 


Limitations 


E. Chalkboard . 
(blackboard) 


1. Inexpensive, can be made locally. 

2. Usable for wide range of graphic 
representation. 

3. Allows step-by-step build up, or 
organization of structure or concept. 


1. Back to audience. 

2. Audience limited to 
50 or so. 

3. Careful drawings erased 
not preserved for future 
use. 

4. (Considerable skill 
required for effective 
use - rarely taught* to 
teachers.) 


F. Flannelboard 
(flannel- 
graph). 
Note most: 
comments also 
refer to 
magnetic 
board 


1. May be used repeatedly. 

2. Usually preparable from locally 
available materials. 

3. Good for showing changing 
relationships. 

4. Hold ^ttention if well used. 

5. Can be adapted for group participation, 


1. For limited audience 
only. 

2. Difficult technique to 
use convincingly. 


G. Field trips 
(not strictly 
media j But 
"useful as 
a comparison 
of factors) 


o 

1. Observation of a participation in 
reality. 

2. Opportunity for cooperative group work 
and sharing "responsibilities. 

3. Good method for individual motivation. 

f 


1. Costly in time and 
transport. 

2. For limited audience 
only. 

3. Requires careful planning 
for effect. 

4. Distractors cannot be 
controlled. 


II. Projectable media 


A, Still pictures 


1. Enlargement of drawn or printed 
materials for large audiences. 

2. Obviate need for producing slides 
and transparencies. 

3. For transferring enlarged image to 
chart or blackboard for copying. 

4. For projection of small objects and 
specimens. 


1. Demands total darkness 
for clear projection 
(except with very 
expensive models). 

2. Bulky machine, 
difficult: to transport. 

3. Electricity required. 


a. Opaque 

proj ection 
(epidiascope) ♦ 
This is 
equipment 
based method 
only as all 
materials 
selected from 
previous 
section I 


b. Transparencies 
for overhead 
proj ection 


1. Projectable in full day light to 
large audiences* 

2. Presented facing audience 

3. Relatively easy to prepare with local 
materials. 

4. Subjects can be drawn in advance or 
developed by stages with the group. 

5. Can demonstrate movements, processes, 
etc. with perspex models. 

V 


1. Electricity required. 

2. Equipment and materials 
for making sophisticated 
transparencies expensive. 

3. Not usually suitable for 
photographic material 
due to Cost (although 
adaptor available to take 
35 mm slides). 

4. Usually restricted to « 
teacher use, as it is not 
easy to adapt for the 
learner to use. 
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Media 


Advan tages 


Limitations 


c. . Slides and 
films trips. 


1. Suitable for large audiences. 

2» Relatively easy production and (in 

black and white) i oduction. 
3. Cheapest current fo of visual. 
4 # Easily adaptable to self- instructional 

units. 

5. Equipment available for viewing or 

projection without electricity source. 

-j 


1. Fixed order of frames 

in filmstrip restrictive 
in use. 

2. Need partial darkness 
for viewing unless rear 
screen or day light 
screen used. 

3. Duplication of colour 
slides expensive (even 
impossible in many 
countries) • 


d. Microfiche 


1. Easy storage and cataloguing of 
large numbers of visuals. 

2. Exchange of information on available 
collections* 

3. Very cheap per image if projection can 
be assured for large groups. 

4. Small and light for easy despatch. 


1. Too small for clear 
naked- eye viewing. 

2. Although inexpensive 
equipment available for 
individual use, large 
group projection 
equipment not readily 
available* 

j . Li.eccr3.ci cy repair ea , 

except when hand viewers 
used. 


B. Moving 
- pictures 

a* Films 

(comments 
include . 
reference to 
both 16 mm 
and 8 mm 
formats) 


1. Close to reality with movement and 
sound. 

2. Suitable for large audiences (16 mm) - 
for small groups only (8 mm). 

3. Compression of time and space* 

4. Emotive, can develop attitudes, pose 
problems, demonstrate skills. 

5. 8 mm loops useful for individual 
instruction. 

6. Good learning source if preceded by 
teacher's introduction followed by 
discussion. 

y 


1. Does not permit self- 
pacing* 

2. Films cC3tly and 
difficult to produce. 

3. Individual films 
relatively expensive. 0 

4. Electricity required. 

5. Equipment difficult to 
transport. 

6. Darkness for viewing 1 
(except rear screen 
ose). 

7. Imported film may 

contain inappropriate 

information (see proviso 

in advantage 6). 

i 


b. Broadcast ; 
(open 
circuit) - 
television 


1. Adaptable to large and small 
audiences in widely distributed 
area. 

2. Capable of gaining and maintaining 
attention* 

3. Can stimulate emotions, build 
attitudes and develop problems. 

4. Can conserve resources of instructors 
by simultaneous broadcast to many 


1. Prograjnme expensive to 
produce and demands 
highly skilled staff. 

2. Receiving equipment 
expensive and difficult 
to maintain* 

3. Electricity required. 

4. No immediate interaction 
or feedback. 

5. Learner must adapt to 
fixed schedule, never the 
other way round. 
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Media 


Advantages 


Limitations 


c. Closed circuit 
television and 
videotape 
^"^{>including 
videb^^ 
cassettes^v^^ 


1, Adaptable to medium and small 
audiences* 

2. Videotape repeatable to fit learning 
schedules* 

3* Film advantages 1, 3 and 4 apply (see 
above). 

-4. Valuable for magnification of image, 
^^r^cording intimate situations, micro- 
teaching, recording of developments 
in clinical syndromes or in 
scientific experiments, "bringing the 
village into the^fcl^ssroom'' , 
recording emenzenciesV-^etc. 

5. Portable equipment can function on 
battery for field recording. \^ 


1. High initial cost of 
production equipment, 
and requirement of 
trained staff, 

2. Electricity required, 
Although portable works 
off battery, this needs 
charging from power 
source. 

p. Receivers are expensive 
and require maintenance. 


HI. Sound media ""X^ 


a. Broadcast 
radio 

* _ 


1. Adaptable to large and small 
audiences in widely separated areas. 

2. Conserves resources of instructors by 
broadcasting simultaneously to many 
classes. 

3. Capable of gaining and maintaining 
attention. 

4. s -Reception equipment relatively cheap 

and will function on batteries. 

5. If combined with prepared materials 
(radiovision) can be improved 
learning tool. 


1. S^eqial studio 

, facilities and staff 
fequirea^or broadcast 
(less expensive and 
complex than broadcast 
television). ;^ 

2. Learners must adapt to -j 
fixed schedule, not 
other way round. 

3. No immediate feedback 
and no audience 
interaction. 


b # Sound 

recording 

(reel and 

cassette 

tape, 

records/ 

discs) 


1. Adaptable to any size of audience. 

2. Especially suited to individual and 
small group learning. 

3. Due to stop%nd playback facilities 
of tape, can be student paced. 

4. Cheap, battery operated cassette 
players available and relatively 
cheap cassettes. 

5. Many uses - to provide sound for 
slide sequences, for micro* teaching, 
heart sounds, for posing problems, 
etc. 


1. Use for individual 
learning demands many 
playback units. 

2. Good quality recording: 
demands studio facili- 
ties. 
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0^/S.T ACLES TO CHANGE 



^ DYNAMIC CONSERVATISM 




INSTITUTIONAL BUREAUCRACY 
BUILT-IN RESISTANCE 10 CHANGE 



^ COMPLEXITY OF CURRICULUM CHANG E 

) LACK OF INSTITUTIONAL RESOURCES 

^ LACK OF MODEL UPON WHICH TO BASE A CURRICULUM CHANGE 



J. BRYANT 
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FOUR C ' S 
OF CURRICULUM PLANNING 



COOPERATIVE 



|^ CONCRETE 

★ 



CONTINUOUS 



COMPREHENSIVE 

/ 
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describes: 

A SERIES OF PLANNED EDUCATIONAL ACTIVITIES 
A STUDENT IS TO GO THROUGH WITH 
THE ASSISTANCE OF TEACHERS < 



INTEGRATION 

( of a curriculum) 

. . . COORDINATION OF DIFFERENT. 

TEACHING/LEARNING ACTIVITIES TO ENSURE THE 
HARMONIOUS FUNCTIONING OF JHE EDUCATIONAL 
PROCESS FOR MORE EFFECTIVE TRAINING 
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* 

PLAN OF* ACTION FOR PREPARATION OF A TEACHING PROGRAMME 

/ « 
















% 


Chrono- 
' logical 
order 


Definition of objective 


Executing body 


Advisory body 


1 

1 


1 


Prospective study to evaluate 
the country's requirements 
(quantitative and qualitative) 
in' respect of health personnel, 
taking into account what is 
available and can be absorbed 


Ministries of 
Planning, Health, 
Education, etc. 


University staff, 
national and inter- 
national experts 




2 1 


l 

Definition of the educational 
objectives of the school of 
health sciences in accordance 
with the tasks to be accom- 
plisneo Dy eacn piuicoo iuii»i 
health worker 


Ministries of 
Education and 
Health 


Special faculty 
commission 

/ 




3 

^_ 


Organization 01 an hi l umiak j.u» 
group on education planning for 
health sciences 


Faculty staff 


Department af f ' 
Education 

Education consultant 




4 


Acceptance Dy cne LdtuiLy ouai-i. ui 
country's requirements; training 


' the goals established in the 
of teachers (education) 


light of the 
% — . 


I 

/, 

** ( 


5 

< 


Modification, of teaching 
programme and selection of 
methods by which to attain the 
goals 


Faculty staff 
(Curriculum ' 
committee) 


Faculty staff 
Education consultant 


6 , 


, Development of « system of 
evaluation: 

— rmai exsminat ion vt.uinuiai.xvc/ 
<- partial examination (formative) 

indicating whether the goal 

has been attained 


Evaluation 
Committee 


Department of * 
Education 

Education consultant 
• 




7 


Organization of a group for 
research in education 


Faculty staff 


Department of 
Education 


\ 


8 * 

M 


* 

Periodic re-evaluation of goals 
and methods in response to 
changes in the country's needs 
and in the light of evaluation 


Ministries of 
Planning, Health, 
Education, etc. 


University staff, 
national and inter- 
national experts 
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Health 
Services 



HEALTH NEEDS 
OF SOCIETY 



EDUCATIONAL 
OBJECTIVES 



REVISION - 
OF 

OBJECTIVES 



SELECTION 



STUDENTS 



1 



TEACHING AND LEARNING 
ACTIVITIES 



E 
V 
A 
L 
U 
A 
T 
I 

0 
N 



GRADUATES 



RESEARCH IN TESTING 
AND MEASUREMENT 



I 



STUDENTS ENVIRONMENT 
BUDGET, HEALTH 
SITUATION, ETC. 




faculty organization 
budget;educational 
competence of teachers 


ACTION ON MANIPULABLE VARIABLES 



WORKING HYPOTHESIS 
CONCERNING CURRICULUM 
PREPARATION 



REVISION OF HYPOTHESES 
CONCERNING CURRICULUM 
PREPARATION 



DIAGRAM SHOWING ^ACTORS 
INFLUENCING CURRICULUM 
DESIGN 
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THE CONCEPT OF INTEGRATION IN EDUCATION AND TRAINING 



] 



■JEFINITION 



Integration (from the Latrt integer * whole) means coordination of different activities 
to ensure harmonious functioning. 

This definition suggests that the result of harmonious functioning will be greater 
efficiency. In our materialistic world efficiency is a dogma. In the developing countries 
it is the key to survival. Neither the idea nor the word is new. At present both are 
fashionable and, as is oftei* the case in such situations, infatuation and passion are more in 
evidence than logical analysis and objective appraisal. Such things are used, abused, trans- 
formed and deformed. 

The password is "integration* 1 . What is integration? 

Let us start by saying that we shall not be considering here "integrated medicine", 
"integration of basic health services" or "integration of the health team". The concept is 
certainly the same but used in the field of a philosophy of public health, of the deployment^ o£ 
services and of human resources. 

We are concerned here with the meaning of integration in the more limited field of 
education and training. To paraphrase the definition given above, integration in this field 
is the coordination of different teaching activities to ensure the harmonious functioning of 
the educational process for more effective staff training. 

2. THE FJRPOSE OF INTEGRATION 



Before the era of the mega-universities with thousands of students. and before the era of 
specializations stemming from a technology ramifying in all directions, medicine was learned 
with the patient in direct contact with the professor - a natural source of integration. 
During training, there were of course several professors and the student, second natural source 
of integration, assimilated, analysed and synthesized the information. / 

* Since the beginning of he twentieth century, the university- factory, the numbered and 
card- indexed student, and the super-specialists have made their appearance. Fundamental and 
staggering scientific discoveries are being made faster than text-books are revised and 
reissued. Research laboratories are draining away and absorbing funds intended' for training 
centres. Teachers engage in research since their future depends on the results they obtain 

their field. Without organized teacher-training, they are entrusted with students who. are 
not supposed to appraise their performance. Reputation depends on the number of publications 
rather- than the quality of the doctors trained. Theoretically tihe doctor, once trained, is a 
complete and integrated element. In practice,, he is the outcome of a non- integrated training 
lacking in human feeling. 

Human feeling, the concept which should inspire students and teachers , should be concergfc^ 
with improving the health of the people and should be the basis of the educational objectives 
of the university centres for health sciences. These objectives, shared by a group of 
physicians with teacher-training, will be mere easily attained if the various teaching activi- 
ties function harmoniously, hence the need for integrated teaching. j 

\ ' 

3. ADVANTAGES OF INTEGRATED TEACHING 
This method has several advantages: 

/ 



. Synthesized presentation 
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In traditional teaching the cardiovascular system, for instance, is dealt with separately 
and sometimes conf lictingly by the anatomist, the histologist, the embryologist , the physicist, 
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the physiologist, the biochemist, the microbiologist, thp. pathologist , the physician, the 
surgeon, the pharmacologist, the medico-legal specialist* and others. Each specialist tackles 
the subject by reviewing what he thinks the others did, for the obvious reason that the 
student has already forgotten it. He treats the subject from the standpoint of his specialty 
with al? the details that it involves. The student listens, repeats and forgets while 
waiting for the next lecture, He has neither the knowledge, the experience, nor the time to 
synthesize the various fragments presented to him, 

The cardiovascular system will be remembered according to the most vivid presentation. 
In integrated teaching, coordination is, carried out^by the teachers in advance. 

Reduction of pointless repetition 

Reminders, over-elaboration and contradictions not only confuse the student but also 
waste his time*. 

Presentation iti a logical order . . 

In traditional teaching, the lack of coordination has often meant that the physiology of 
an organ is dealt with- one year and its anatomy the following year. Or the clinician presents 
the etiology of a disease before^ the microbiologist has been able to described the organism 
responsible for i£. 

~" * ■ ^ * 

Reduction of non-essential details 
/ 

Preliminary discussion by the different teachers^ of a specialty automatically results in 
the pruning and balancing of a subject in line with the comnwn objective. 

Improvement of the quality of peaching and teachers 

This is achieved, if for no other reason, by the process of emulation which always occurs 
when several specialists who are normally isolated find themselves working side by side. 

Improvement of relations between the departments through preparatory meetings 

Better utilization of teachers who are nd longer restricted to their own departments but 
can assist with a teaching fuqction whenever they have the necessary capacity. 

4 # A PPLICATION OF THE 'CONCEPT OF INTEGRATION ACCORDING TO THE TEACHING METHOD 
; 7 : 

Theoretical instruction 

At the outset, let us exclude from our discussion the traditional lecture. It should be 
a collective information session on a subject not dealt w^th in texts. Then it may have a 
stimulating and catalysing quality. As it is it belongs to the realm of propaganda, magic and 
exorcism, not of integration. 

However, discussion in small groups on the lines of the tutorial, accompanied by a case 
study, presentation of patienLs or a demonstration, clearly corresponds to the concept of 
integration. 

* The same is also true of clinico-pathological or clinico-social lectures. 
Practical instruction 

Clinical practice at the patient f s bedside is a form of instruction which also benefits 
from the presence of different specialists (clinician, epidemiologist and basic sciences 
specialist) . 

4 
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Laboratory exercises. This is certainly the field in which integrated teaching has made 
its greatest contribution in the form of a multidisciplinary student laboratory. It is in 
such laboratories that' the basic principle of the Western Reserve reform had its best practical 
application - teaching centred around the student. The earliest plans date back co 1948, 
Since then many medical schools (almost all those planned after 1960 in the United States of 
America) have set up suitably adapted laboratories. 

* The essential aim is as follows: to give each student an individual work area in which ^, 
all practical laboratory work is done. The student is also giver his own study, which is 
open to him 24 hours a day, in which he can keep his books, microscope, etc. This is a 
facility that was previously limited to postgraduate students. Finally, the aim is to ensure, 
from the standpoint of laboratory staff and logistics, economic coordination and centralization. 
The students are generally grouped in modular units of 16 places as in the universities of 
Brasilia, Southern California, Florida and Kuala Lumpur, although this arbitrary figure is 
neither historically based nor essential. It is also possible to establish units of eight 
students as at Einstein or of 24 as at Harvard: in the otiginal multidisciplinary laboratory 
at Western Reserve there were 80 students in one room. Aesthetic reasons and the desire to 
give the student a less community-type atmosphere have in most cases led to the choice of 
modular units of 16. The teachers in the various fields come to the laboratory, according to 
an established programme, to present the exercises as scheduled, using apparatus set up in f 
advance. The multidisciplinary laboratory is ..run by a director responsible for coordination 
withxthe different departments concerned. He is generally a basic sciences specialist chosen 
for hiX tact and wide interests. :* 

Experience in ,the matter has amply demonstrated some of the advantages of mtjCLt id iscip Unary 
laboratories over departmental laboratories. The students appreciate haying a place in which 
to^wotOc in the right atmosphere. The teachers recognize that their teaching h;s greater 
effectiveness in less space and at lower cost. In short, the advantages of the multidiscipli- 
nary laboratory aire that it: 

- gives the students a fixed working' base; 

- eliminates movements of students from one laboratory to another; * 

- makes for effective teaching; 

- permits a saving in building space; 

- avoids duplication of equipment; | 

♦ 

- permits more rational use of teachers. 
5. STAGES AND PRECAUTION S 

Universities which have succeeded in applying a system of integrated teaching have done so 
by stages. ^ 

However sound an idea, for successful implementation it must be presented, tried out and 
defended by someone who is determined, competent and sufficiently diplomatic not to make more 
enemies than friends.- 

The administrative structure must be modified so as not to present any obstacles to the 
concept of integration. 

< 

A committee (department or unit) of information and research in medical education is an 
essential instrument for promoting the concept of integration. 
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There should be total separation between the planning of teaching on the one hand and the 
allocation of funds (general or research) and appointment of teachers on the other. The 
traditional view that the more teaching hours' a department ha's the more important it is (and 
hence deserves more staff and funds) leads to anti-educational decisions. 

6. CONCLUSION 

The advantages of integrated teaching(specially if competency based) seem su evident that 
they raise the question why, after 30 years of positive experience, it is still necessary to 
promote it so strongly? If it is so good, why have all the new medical schools not joined in 
the movement? Why have the older medical schools not modified their curricula in favour of 
an integrated system? One answer is that in the university world in general and the medical 
sector in particular, individualism, academic self-sufficiency, caste attitudes, a superiority 
complex and conservatism are stronger than the desire for coordination, concerted action, 
participation, modesty of outlook and new ideas. Mental inertia is as difficult. to overcome 
as all types of inertia. We shall have to rely on patience, optimism and the slow, long 
training of those who will eventually take over. 




/ 
/ 

THE MAIN PITFALL OF THIS CONCEPT IS THAT IT IS TEACHER ORIENTED , 
IF YOU ARE MORE INTERESTED BY A STUDENT ORIENT EC? CURRICULUM, 

SEE PAGES 245-252 
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PLANNING CURRICULAR CHANGE IN ORDER TO INTRODUCE 
AH INTEGRATED CURRICULUM 



] 



I! PREFACE 

A key component of educational reform and innovation is the organization modification 
needed'to plan and implement the desired changes. In other words, how will the organization 
be modified to facilitate change? Who is to do what and in what sequence? Is the 
completion of some tacks pie-requisite to beginning others? What supporting actions need 
to be taken and what resources need to be made available in order for the change to take 
place in an orderly and effective fashion? m , 

In order to illustrate some of the planning principles which need attention, the 
attached planning s chedule has been prepared as an example. 

2 . Why a planning schedule 

It is designed to help to answer the following questions: 

2.1 If a curriculum is designed on the basis of objectives (compete ncies) to be 
achieved rather than time to be used , how should this affect the planning process? 

,2.2 How can a faculty be organized for an "integrated" curriculum? 

3. Planning schedule concepts - % 

The concepts both explicit and implied in this plan, represent a radical departure from 
traditional medical school practice. If a school were to shift to the type of system 
described here, the transition should be regarded as a long- term (perhaps four or five years) 
project. This suggests that: 

3..1 A plan for phasing from the old system to che new one should be developed. 

3.2 A series of intermediate objectives should be set with time deadlines. 

3.3 The objectives should indicate what parts of the old system should be improved, 
which elements of the new system should be introduced, and in what sequence. 

3.4 - The 'plan should be discussed with the faculty and appropriately revised before 
becoming institutional policy. 

4. Sequence of steps 

The system proposed is hypothetical and should be used only as a means for beginning 
work on a system appropriate for your own school. 
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A representative sequence of steps for planning 


and implementing an 11 integrated" curriculum 




ACTION 


TAKEN BY 


OBTAIN AGREEMENT FROM 


1* Curriculum Committee appointed with 
authority and responsibility to plan and 
coordinate an integrated curriculum. 
Members should be selected for creative 
abilities as we'll as for distribution among 
disciplines* Students and perhaps a 

licaxtu lliJ.llJ.aLLy LcpLcocllLdLl Vc SuOUlQ/ Dc 

included* 


7 

Medical school 
administration 


University administra- 
tion; major power 
centres in medical 
scnool 


2. Institutional objectives are stated in 
writing in terms of the abilities and 
attitudes graduates should posess. A 
s LaLaucut uj. gcucidx cumcuxar pnnusopny 
should be prepared* 


Curriculum 
Committee 




3* Approval of written institutional 
objectives and curricular philosophy are 
obtained* 

Revisions, made as needed to obtain agree- 
ment' with the plan in. principle. " 


Jointly by medical 
school administra- 
tion and curriculum 
committee 


University adraini3tra- 
tion; major power 
centres in medical 
school; Health Ministry. 


4. Select the organising principles that 
- T will be used for planning the integrated 
curriculum* Examples are: 

- competency base; 

- body systems in health and disease; 

- clinical problems base; 

- the processes of clinical medicine; 

- a preventive medicine models; etc* 

- community problem base 

5* Lay out a general sequence of curri- 
cular units in block diagrams or outline 
form based on: 


Curriculum 
Committee 

Curriculum 
Committee 


Medical School 

A rim i n l ft t*T"£i t*i on 

flUlll J. 11 J. O L. L Cl L. J.U 11 


5*1 Curricular organizing principles 
and philosophy; 






5*2 Assumptions and data concerning 
incoming capabilities of students; 


I 




5*3 Terminal curricular objectives; 






5*4 Available instructional resources. 






6* Select the disciplines that should 
have major representation in each curri- 
cular unit. ^ k 


Curriculum 
Committee 


*3 
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ACTION 



Organize Teaching Committees for major 
teaching units. Critical factors: 

7.1 Teaching unit chairmen who: 

(a) understand and support 
the philosophy of the curri- 
culum; 

(b) have demonstrated ability 
to get things done in coramit- 
-tees. ' 

7.2 Majority of committees are 
open-minded and willing to break with 
tradition. 

7.3 < Major disciplines specified in 
step 6 are represented. t 

8. Select teaching unit faculties. 

9. Begin training sessions for N 
Teaching Unit Committees and Faculty, 
General Topics: 

9.1 Philosophy and objectives 
of curriculum; 

9.2 Educational concepts and 
methodology; 

9.3 Organizational principles 
essential to the success of the* 
curriculum. 

10. Teaching unit pre- requisite abilities 
and terminal objectives (competencies) 
(department level) are prepared or 
reviewed by Teaching unit committee. 
Preferred sequence: begin with last 
teaching unit prior to graduation and 
work backward to entry point in 
curriculum. 

11. Teaching unit objectives (or plans) 
are submitted to review and revision 
process. 



12. Instructional objectives are 
prepared and reviewed. 



TAKEN iY 



Curricu 



lum C 



ommittee 



Unit committees 

Curriculum 
^Committee - with the 
assistance of educa- 
tion consultants 
from: 

(a) own university 

(b) other organi- 
zations 



Teaching uhit commit- 
tees with guidance 
from curriculum 
committer/ and educa- 
tion * specialists 



Curriculum Committee, 
school administra- 
tion, education 
specialists 

Teaching unit 
faculty, unit 
committee, education 
specialists 



OBTAIN AGREEMENT FROM 



Medical school admini- 
tion; departmental 
chairmen 



Departmental chairmen 



\ 



Department chairmen 
concerned 
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ACTION 


TAT IN BY 


OBTAIN AGREEMENT FROM 


13. Instructional experience and evaluation 
methods (both immediate and long-term) are 
planned. Necessary administrative support 
laencxriea ana pxannea. 


Unit faculties under 
guidance of unit 
committees and edu- 

Lu L J.ULL O^CtidiiO L9 


Curriculum Committee, 
chairmen concerned, 
school administration 


14. Instruction and evaluation methods t 
implemented.. 


Unit faculties under 
guidance of unit and 
curriculum committees 


Department chairmen 


JLj» LValUaClOu reSUXCS t-UXlct-LCUi L/d La. 

pertaining 4 to individual student evaluation 
and evaluation instructions handled 
separately and processed through appropriate 
channels. 


Unit* r* n mro i t* t* p p anH 

education specia- 
lists 




16. Recommendations for improvements in n 
the system are developed based on evaluation 
data and other sources of information and 
advice. 


Curriculum Committee, 
unit committees> 
school administra- 
tion 


Appropriate individuals 
and groups 


17. «, Changes are implemented at appropriate 
points in the system. 


Curriculum Committee, 
unit committees 


School administration, 
department chairmen 



\ 

★ ★ ★ 
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5. Possible advantages of a systematic approach ' 

5.1 It provides a mechanism making possible a study planned and organized in advance. 

5.2 When it becomes really operational it "should be more effective than the 
traditional approach since it can be improved by' the feed-back. 

' 5.2.1 If it is well organized, it permit greater flexibility in organizing 
the use of time of students and teachers. 

* 

5.2.2 Certain individual variations can be foreseen in the rate and mode of 
progress of students throughout the curriculum. 

5.2.3 It may prove possible to train more students at the same cost if the 
system is carefully structured. 

6. Possible disadvantages 

6.1 Teachers must spend more time on planning and evaluation, especially during 
the first years following the adoption of the system. In many respects it is more 
expensive at the outset to try to construct a new system rather than to keep to the 
old one. 

6.2 It seems clear that a nucleus, of teachers will have to spend a greater part of 
their time in planning, managing and evaluating the curriculum. These activities could 
be organized^ ing a rotation system covering three or four years, so as to distribute 
the work and give teachers an opportunity to become familiar with the methodology. 

7. Necessary supporting elements ^ 

* . 

7.1 A school administration and departmental chairmen willing to delegate some- of 
their responsibility and authority to a new curricular planning and management system. 

7.2 Budgeting of funds in a manner consistent with the curricular plan and the 
distribution of decision-making authority. 

7.3 Review and, where necessary, revision, of academic reward policies of the school 
so that faculty contributions to planning and implementation of the curricular system 
are given high priority when decisions are made concerning salary increments and 
academic promotion. 

** 

7.4 A medical school administration and curriculum committee who are skilled in 
'co-opting, "neutralizing" or "removing" key opponents to the plan on the faculty. 



This procedure for introducing integrated teaching was prepared (1971) by Dr J*. Hess, 
Wayne State University School of Medicine, Detroit, Michigan, United States of America 
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ORGANIZATIONAL STRUCTURE ASSUMED FOR SIMPLE CURRICULAR 

PLANNING MECHANISM 



Administrator in 
charge of curric. 



School 
Administration 



Education 
Consultants 



School Curriculum 
Committee 



1 



_ J 



School 
evaluation committee 



Year 1 
Coordinator 
and Committee 



Year 2 
Coordinator 
and Committee 



Year 3 
Coordinator 
and Committee 



Year Nth 
Coordinator 
and Committee 



□ 



□ 



Teaching Units 



Key: Sojid lines represent direct lines of administrative authority. 
Dashed lines represent advisory authority. 
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Personal notes 
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EXE R CISE 




\ Show diagramatically the type of organization (commissions, committees, boards, etc, 
indicating their functions) which you deem feasible (in the establishment where you are 
teaching; in order to introduce (or improve) integrated teaching for a competency based 
curriculum . . . . then compare with ^age 24 1. 
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E X E R C I SE 



Describe the constraints which you are liable to encounter in implementing the 
organizational" plan you have proposed on the preceding page, showing for. each constraint 
the STRATEGY for overcoming it: 
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A LKARNKR .OR I BNTKI) CURRICULUM with built-in RKLKVANCE 



In the- teaching of health sciences an essential quality of the programme - namely its 
relevance - has often been neglected, by educationalists. The relevance of a~ programme is the 
extent to which it , facilitates the acquisition of the level of competence and areas of skill 
necessary for the student to practise his profession. In chapter 1, the basic elements^ 
necessary for formulating relevant edutational objectives were indicated. It was then 
stressed that the health needs of populations, together with their available resources, should 
serve as a basis for such formulation. A further point made^was that health planners should 
define the types of health service organization most suitable' for resolving the health problems 
of a given population as it is within the framework of these services that the individual 
members of the health team wiH play their roles, fulfil their functions and accomplish their 
special tasks. *These roles, tasks and functions must also be defined not only in terms of 
the present expectations and needs of the communit> but also in terms of the likely future 
roles, tasks and functions which members of the liealth team will be expected to fulfil in the 
future. Consequently prospective definitions are required, such forecasts being dependent 
on the complexity of the abilities to be acquired and the duration of the training programme, 
||or those professions requiring high levels of competence (physicians, sanitary^engineers , etc.) 
it has not been possible to define the tasks they will have to carry out^in 10 years, other 
than in the form of assumptions which have variable degrees of reliability. Perhaps the 
developments in the science of futurology will ultimately end in finding solutions for this 
problem. 

Until this is possible, however, other solutions should be envisaged. ^ 

A few years ago Laurence Weed suggested a new programme of medical education which attempts 
to resolve the notorious and difficult problem of relevance. The orginality of Weed's proposal 
is that it seems to resolve this problem while remaining within the strict framework of 
training* (refer cnce more to the diagram on page 114) and without making use of prospective 
data on tjie organization of the health services. It is for this reason that ve are referring 
to it in this Educational Handbook. We also suggest that die reader consults other specialized 
works concerned with alternative methods of programme construction. 

We shall propose a type of educational programme^ inspired largely by that of Weed blit 
comprising an orientation closely linked to community rather than individual health problems. 

At the outset, it should be mentioned that the definition of educational objectives (at 
the most general level) should, in all cases involving the training of higher staff, include 
the following objective: "be able to find solutions for problems ^rising in new situations". 
It is here that the basic sciences have an important role to play, assisting the student in 
the acquisition of a method of work essential tor a graduate if he is to keep his knowledge 
and skills up to date. 

Weed's pew programme 

The Weed programme based on a certain number of premises which must be stated first of 
all, since any disagreement with these premises would probably lead to a rejection of the 
proposed programme. On the other hand, the mere fact of agreement with the premises does not 
ensure that the teachers would be willing to accept and apply the new programme since Weed's 
approach is revolutionary: it requires replacing a whole programme not merely changing parts 
of it. 



1 The proposed method can be adapted just as readily to the preparation of the programme 
for a dentist or a nurse and, by extrapolation, to that for a sanitary engineer, pharmacist or 
hospital administrator. 

Described in his book: Medical Records, Medic* i Education and Patient Care. Cleveland, 
The Press of Case Western Reserve University, 1970. 
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The problem is that . the premises seem relatively harmless and such that the most con- 
servative of teachers is prepared not only to support £hem but even to say that they contain 
nothing new and that he adopted them a long time ago, or possibly that he has thought of them . 
himself. 

Weed's premises 

1. General educational objectives. The student (future physician) must be able to : 

1.1 set his personal goals related to his work within the organized system of medical 
care, and to develop his own appr^ches to the achievement of these goals; he must also 
seek criticism from teachers and fellow students. 

1.2 cpllect relevant data and utilize that for solutions of the problems. 

1.3 audit his own performance or that of a colleague using carefully established 
criteria. 

1.4 demonstrate the qualities of professional thoroughness, reliability, sense of 
analysis and efficiency. . 

</ 2. Teaching and learning techniques : * 

2.1 The student must acquire basic skills (manual and intellectual) ,and not merely 
basic knowledge; he must be able to demonstrate that he can do things; the tyrannical 
role of memory must be reconsidered; long-term memory is usage. 

2.2 He must be able to learn and think independently and to learn from his experience. 
Every patient can be a source of data, while the teachers should^ ensure intellectual 
discipline and value judgement. 

2.3 He should be able to acquire independently, thanks "to study of documents and 
information prepared for the purpose (books, scientific journals, duplicated texts, 
audiovisual aids, etc.), everything that is usually transmitted by traditional 
lecture-courses, provided that the educational objectives are known to him and 
formative evaluation exercises constructed on the basis of the objectives. 

i 

2.4 He should acquire his professional competence through the practice or real 
; professional acts of increasing difficulty, compatible with his level. 

2.5 Practice in a controlled environment may lead to the revelation of basi\ principles . 

2.6 A function of the teaching institution should be to help students find solutions 
for a diversity of new problems and make them able to carry on once the problem to be 
solved has been defined. 

3. Organization of studies : 

3.1 The quality of achievement and acceptable performance for a given tasjc should be 
* regarded as a ccnstant. What can vary is the number of types of tasks fchat a given 

health professional should be able to accomplish. > 

3.2 The length of study programmes should not be regarded as a constant. It is 
preferable to have a clear definition of the performance to be acquired, and for the time 
necessary to master thorn to vary for different students, depending on their ability, and 
drive, the availability of teachers and educational facilities. 
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3.3 There is no natural frontier in medical practice between the physical sciei.ces and 
the humanities, nor between science and culture in the broa lest sense of the word. The 
giving of courses in the humanities and the behavioural sciences does not automatically 
ensure that future physicians will be, able to relate their daily practice to the culture 
to which they belong. On the other nana*, they will do so if they are required to define 
the patients 1 social and psychological problems in addition to his medical ones, and to 
look for a logical solution, either by themselves or with the aid of specializes. 

3.4 The educational role of the basic scientist is not to train the future physician in 
his academic specialty but to ensure that the s tudent is able to detect any violation of 
basic scientific principles in his practice . 

Features of the new programme : COMPETENCY based CONTRACT . 

The programme includes 4 phases at the end of which the student is awarded a degree 
certifying he has a satisfactory level for specified competencies. 

The total duration of each phase may vary for each student and is determined by^the 
achievement of an acceptable Ifevel of performance for a given number of educational objectives 
which constitute, as a whole, the basis of a contract between the student and the teaching 
institution. Dependir ; on the terms of the contract, the degree may be that of doctor of 
medicine, medical assistant or specialist in cardiology for example. 



1.1 Show competence in planning education. 

1.1.1 Define educational objectives in terms of behaviour. 

1.1.2 Plan a learning programme. 

1.1.3 Construct tests and other measuring systems. 

1.2 Utilize the resources of the institution, namely: microscopes, slides, library, 
computers, members of the health team. 

1.3 Describe the concept of health tealn? 
Second phase (approximate length - 12 to 18 months) 
2. At the end of the second phase, the student should be able to: 

2.1 Prepare a list of what he considers the ten main health problems of the community. 

2.2 Collect data "base" relevant to the community and interpret these data in order 
to . . . 

2.3 . . . identify its priority health problems. 

2.4 Describe health activities which would enable these community health problems to be 
resolved and compare these activities with those usually undertaken. 

2.5 Identify the professional role which he (the student) wishes to assume at the end 
of his basic training. 



General educational objectives of the new programme 



First phase (very approximate length - 1 to 3 months) 



At the end of the first phase, the student should be able to: 
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2.6 \Describe the tasks corresponding to that role. * 

2.7 Prepare educational objectives (with evaluation criteria) whereby acceptable com- 
petence can be achieved for each of these tasks. 



It is THEN that the CONTRAC'l is signed between the student and the university. 



Third phase 1 (very approximate length - 1 year) 

3. At the end of the third phase, the student shouK be able to: 

3.1 Collect the f, data base" on the members of the community, both sick and healthy 
(th" meaning of data base: chief complaint, patient profile, systems review, physical 
exa. '.nation, laboratory data, etc. is clearly defined by Weed; the reader is advised to 
consult his works and make sure that students collect all the necessary data). 

3.2 Audit the data base for a given patient collected by another student. . | 

3.3 Carry out clinical techniques ( such as phlebotomy , sutures, lumbar' puncture, fundus- 
copy, auscultation, etc.) corresponding to the educational objectives described under 2.7. 

Fourth, phase (very approximate length - 3 years) „ * 

4. At the end of the fourth phase, the student should be able to: 

Carry out (together with other members of the health team) all tasks involved in the 
health promotion/rehabilitation of the community in which he is living. 



Note - It is obvious that there exists, and will continue to exist a relationship between 
the socio-political system of the community and its types of health and educational 
services . 

Moreover, since student selection criteria also reflect the socio-political system, 
it should be borne in mind that they can influence the results of the training 
programme. 

Finally, it would appear that this approach can be utilized in an y kind of country, 
whether poor or rich. While the concept of data collection, and evaluation 
systems/ suggest that computers could be usefully employed, they are not essential. 
Recourse to data processing and the associated equipment is often useful and some- 
times indispensable when a large number of data have to be managed, but it is not 
a neutral instrument and in the developing countries care should be taken that data 
processing does not represent an instrument of economic and technological, and 
consequently, political , dependence. 



1 The third and fourth phases correspond to the programme in the form proposed by L. Weed 
in The New Curriculum, the problem-orientated system , Medcom 1972, pp. 95-104. 
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1 

The new role of the teacher 



(1.1) 1. To be available to provide students with a constructive critical 

analysis of their working objectives and methods. 

(1.2) 2. To be competent to analyse and evaluate health problems identified by 
^ students, as well as p^jposed solutions. 

(3.2) 3. To define the competencies which students should ke encouraged to acquire 

in order to solve a given health problem. 

(1.3) 4. To construct criteria in order to facilitate the assessment of students* work 

and to measure their competencies. 

I 

(2.4) 5. To select professional activities of graded difficulty appropriate to the 

*' students' level during the whole of their learning curriculum. 

-(2.3) 6. To prepare working documents and provide information which would facilitate 
students' independent work and to evaluate the efficiency of such documents. 

(2.6) 7. To confront students with a diversity of new problems in order to help them 
to define their parameters and find solutions to them. 

; (2.1) 8. To facilitate students' learning by putting questions requiring problem- 
solving skills rather than recall. 

(2.5) 9. To verify that the students find the basic principles underlying the professional 

activities which they are required to accomplish during their cycle of studies. 

(2.2) 10. To ensure intellectual discipline by careful observation of students' work and 

to make a value judgement on their activities. 

(3.1) 11. To be uncompromising in relation to students' performance level for a given task. 

(3.4) 12. To verify each step the student takes in solving health problems, in order to 
identify the possible violation of basic scientific principles. 

(3.3) A3. To verify that the student ..identifies and takes account of bio-psychological, 

cultural and socio-economic factors underlying the health problems which he is 
required to solve during his course of study. 

(1.4) 14. To serve as a role model and give an example of reliability, sense of analysis, 

efficiency and proiessional ethics. 



The numbers in brackets correspond to those of "Weed's premises'*, pp. 246-247. 
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Carefully read {or re-read) Weed f s 14 premises (pp. 24b and 247) and circle the numbers 
of all those with which you agree. * 

If you agree with at least ten of themt read "the new role of the teacher" and circle the 
numbers of all the roles you think yuu can already assume, taking into consideration present 
academic regulations, your personal situation and other local constraints. In each Case 
describe in CONCRETE terms what activity you would perform during the next 12 months. 




*If you do not, try to discuss the matter with another colleague and see if it helps 
u to modify your opinion. 

ERIC 
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Exercise 



/ 

If you are of the opinion that all of Weed's premises are desirable , try to go 
further and describe all the obstacles you would be liable to encounter if you were to try 
to introduce the competency based' programme (pp. 247 and 248) or some adaptation which you 
might prefer, indicating the tactics for overcoming each obstacle. 
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QUALITIES OF A . CURRICULUM COMMITTEE COORDINATOR 



He need not necessarily always have all the following qu&Hties, but if you had to 
choose. at some time or* another between two colleagues it is suggested that you choose the 
one* mojst resembling, this description. 
He should be" AVAILABLE 
^MOTIVATED 

HAVE THE SUPPORT OF THE SCHOOL ADMINISTRATION 
ACCEPTED BY THE STUDENTS 

anci also have as many as possible of the qualities listed below: 

> 7 

1, Very thorough knowledge and wide 'professional experience 

2. Recognized skill in educational planning 
3* The personality of a pleader 

4* A reputation as a research worker * 
' 5* Years of experience as a faculty member 
6* A reputation as an author 
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CURRICULUM SPECIFICATION TABLES 



, — These J 1 tables"* are, so to speak, check lists enabling you to make sure that the ^ 

curriculum will really be prepared taking educational objectives into account, for it may 

happen that the best-intentioned teachers can, on the one hand, make an effort to define 

f. » 

educational objectives, and, on the other, then prepare curriculum which, on analysis, 
seem to neglect them completely. So as to "force" you to bear these, objectives in mind 
the following specification tables have been prepared; they will help you to relate each 
-part of the curriculum directly to a given objective. 

In other words "every paiw of the curriculum should correspond to an objective" and 
"every objective should have a corresponding element in the curriculum". 



* 



Q SPECIFICATION TaBLES ARE AN INSURANCE AGAINST NON-RELEVANT PROGRAMMES ^ 



The same principle -an be employed to "check" an examination system (see page 454). 
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EXAMPLE OF CURRICULUM SPECIFICATION TABLES 
- Public Health Administration - 



ADMINISTRATION 
(Institutional obj. 2) 


Theory 


Problem Solving 
(case study) 


Total 


General administration (Intermediate ob j . 2.1) 


10 


20 


30% 


Planning (intermediate obj. 2.2) 


5 


15 


20% - 


Research (Intermediate obj. 2.3) 


5 / 


20 


25% 


Statistics (intermediate obj. 2.4) 


5 


20 


25% - 


Total 


25 


75 


100% 



X V 



GENERAL ADMINISTRATION^ 
(Intermediate obj. 2.1)-- 


Cognitive 
domain 


Affective 
domain 


Psychomotor 
domain 


Total 


Analysis .of administrative situations 
» (Specific obj, 2.1.1) 


10 


15 


5 


30% 


Human needs versus bureaucracy 
(Specific obj. 2.1.2) 


5 


10 


5 


20% 


Administrative decision making 
(Specific obj. 2.1.3) 


10 


5 


5 


20% 


Group "dynamics, Team work 
(Specific obj. 2.1.4) 


5 


15 


10 


30% 


Total 


3(P 


45 


25 


100% 



* The absolute numbers indicate hours. 

** The numbers of hours has been multiplied by 10 in order to facilitate understands 
of the table. 
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And now ). . use a specification table for your own teaching programme (or part of): 



TOTAL 



TOTAL 



100 7. 



137 



PROGRAMME, 



259 



r 



> A 



. IT WAS FELT USEFUL AT THIS MID-POINT OF 
THE EDUCATIONAL HANDBOOK TO ATTEMPT 
TO SUMMARIZE THE MAJOR CONCEPTS PRESENTED SO ; 
FAR AND PLACE IN PERSPECTIVE THE TWO FOLLOWING 
CHAPTERS. WE FELT THE FOLLOWING ARTICLE 
WOULD VERY WELL SERVE THIS PURPOSE. . * 




YOU WILL FIND LATER THE NAME OF THE AUTHOR AND, cWHAT IS MORE 
SIGNIFICANT, WHEN IT WAS WRITTEN. 



if ■ 



138 



ERJC 



260 k PROGRAMME 



Distinctive attributes of education for the professions 



1 



/Educational ferment characterizes schools and colleges throughout the world. New 
responsibilities, new definitions of educational aims", new programmes are emerging at all 
livels of the educational ladder. At first glance, these new developments appear to have 
no coamon elements but to be highly individual, unrelated incidents of educational change. 

It is helpful to consider first the essential characteristics of a profession and, 
second, the major tasks involved in planning and conducting an education al programme. In 
the light of these two factors, the significant elements of effective education for the 
professions can be Seen more clearly. 

A profession based on a code of ethics 

i 

From the standpoint of the education required, there are two essential characteristics # 

of a. true profession. The first is the existence of a recognized code of ethics. This 

ethical code conmits the members of the profession to certain social val ues above the sel'fish 

ones of income , power, and prestige . In the case of medicine, for example, its code of 

ethics dedicates the doctor to the saving of lives and the protection of the patient's health 
above all material and personal considerations. 

A professional code of ethics not only professes social values that are above selfish 
ones, but it expects the individual member seriously to dedicate himself to these higher 
values . Furthermore, a profession establishes some form of group discipline in support of 
these values. A^ doctor who violates the ethical code of medicine receives the disapproval 
of fellow doctors and is subject to discipline by the state for malpractice. One of the 
indications that an occupation is becoming a profession is a concerted movement among members 
of the occupation to establish and maintain group discipline in order to uphold the ethical 
values tp which the group gives lip service. 

A common problem in several professions is to distinguish ethical values that are broadly 
social from a pseudo-ethical code that in reality dedicates the individual to the selfish 
interests of the group rather than to mankind generally. % In some programmes for the education 
of doctors, there has been a tendency to develop a code of ethics in which the doctor dedicates 
himself to the medical profession and to the interests of doctors rather than to the improve- 
ment of the health of mankind. 

A profession is not a union. Professional ethics must focus upon higher social values I 
than dedication to the selfish interests of the professional group. Yet to attain this high 
aim is one of the serious problems of the professions. 

A profession based on principles i 

The second distinguishing feature of a profession is the basing of its techniques of 
operation upon principles rather than rule-of-thumb procedures or simple routine skills. 
For an occupation to be a profession it should involve complex tasks which are performed by 
artistic application of major principles and concepts rather than by routine operations or 
skills. This is an important differentiating feature. A skilled trade, obviously, involves 
some fairly complex tasks, but the members of the t f ade are able to perform these tasks through 
acquiring certain routine skills and through following certain specified rules. Many of the 
problems encountered by a member of a profession are in a certain sense unique. To solve such 
a problem he must draw upon certain basic principles. However, the application of these 
•principles necessitates an analysis of the particular problem to see what are its unique 
aspects which will require adaptation of the principles. This adaptation is an artistic task; 
that is, it involves individual judgement and imagination as well as skill . A skilled trade 
does not-demand this type of intellectual operation. 
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In the early days, surgery was not really a profession but was a skilled trade. Certain 
skills, such as those used in bone-setting, were transmitted from one generation of surgeons 
to another, and the surgeon learned largely as an apprentice how to carry on his trade. With 
the development of the basic medical sciences like anatomy and physiology, it became possible 
to gain a'more fundamental understanding of what was involved in bone structure and in tissue 
development, so that a surgeon with adequate scientific background was able to adapt his 
particular procedures to the specific conditions surrounding a g iven case. He then solved 
the problem in each case in terms of basic principles rather than simply following rule-of- 
thumb procedures, 

" Whenever a member of any profession meets his day-by-day tasks in terras of routine 
performance, Vf or him the occupation is no longer a profession. 

A profession not only utilizes basic principles rather than depending on rule-of-thumb 
.procedures, but as it becomes more mature it recognizes that the principles used in the 
profession must be viewed in an increasingly larger context and that, correspondingly, the 
science needed by the profession must be continually extend ed to more basic content rather 
than restricted only to the obvious applied science . Thus, increasingly has mWdicine come 
to recognize the interrelationship of nutrition, physiology, anatomy, biochemistry, and 
other fundamental sciences which give a much broader basis for understanding a particular 
medical condition of a given patient . 

In general it can be said that as a profession becomes increasingly mature, it not only 
develops members who carry on their work through principles Ser than rule-of-thumb 
procedures, but it also encourages members to gain an unders- iing oF these principles in 
a much larger context than that afforded by the usual confine s of the occupation. 

. X I have emphasized these two major characteristics of a profession - the development of 
a code of ethics, and the use of techniques that are based on principles - both because they 
a?e the most significant differentia of professions from other occupations and also because 
they help to indicate some of the fundamental tasks of professional education. 

Tasks in planning and conducting an education al programme 

Shifting our attention from the essential characteristics of a profession to the tasks 
involved in planning and conducting an educational programme, we find it possible to classify 
these tasks into four major types, namely: (a) deciding on the objectives; (b) selecting 
learning experiences that will contribute to the objectives ; (c) or fi anizin 8 the learning 
experiences to maximize their cumulative effect; and (d) ev aluating the effectiveness of the 
educational programme in attaining its objectives through appraising the educational progress 
of the students . 

Education is a pro cess for changing the behaviour of student s in desired directions. 
The term " behaviour" is used in the br6ad sense and includes thinking , feeling, and acting. 
When a student is educated he has acquired ideas, habits, attitudes, interests, ways of 
thinking, and professional skills which he did not have before he went to school; his 
behaviour has changed. 

From this definition of education it is clear that the educational objectives are the 
behaviour patterns that the school tries to develop in the student . The knowledge, the 
skills, and the ways of thinking that the student is expected to acquire are examples o^ 
these objectives. The task of deciding on objectives is an important one because these 
the aims and purposes which can and should be used to guide the entire educational programme. 
The only rational basis for selecting learning experiences a nd devising evaluation procedures 
is in terms of their relation to the educational o bjectives. 
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khe second major task of education is to select learning experiences that will contribute 
to the objectives. This raises a previous question: how do people acquire these desired 
changes in behaviour patterns? " People acquire them by practicing them (! is a simple but 
fairly accurate answer. Getting students to practice the desired behaviour, however, is not 
*^the easiest thing in the world* 

A student develops understanding by recalling ideas, by explaining them in his own words, 
and by finding illustrations of them. Skill in ways of thinking are developed by practicing 
problem-solving again and again . Manual skills and habits are also acquired by practice . 
An attitude is acquired as the student looks repeatedly at the phen ome non from a new perspec - 
tive . Interests are acquired begetting satisfaction from certain kinds of experiences so 
that the experiences become increasingly satisfying. For all of these kinds of behaviour, 
students' acquire new behaviour patterns by practicing them * 

One fact clearly emerges from this analysis - the teacher cannot learn for the student * 
Whether or not the student develops an understanding will depend upon what is go^ng on in his 
mind, not what is going on in the teacher f s mind. To plan learning experiences is to outline 
the activities that will give the students a chance to practice the behaviour implied by the 
objectives. Thus, planning a particular course will mean providing situations in which 
students will encounter problems to solve so that they can gain understanding and develop 
critical thinking . Planning may also involve providing tasks which require the students to 
practice manual and intellectual skills and habits, and so on. * 

The third major task in education is to organize the learning experiences to maximize 
their cumulative effect . We can increase the cumulative effect of learning experiences in 
two ways: through sequential organization and through integration. When learning 
experiences are arranged so that the students begin with simpler concepts and skills and go 
on to b*-~ and deeper applications, far greater learning results than with a random arrange- 
ment of learning. This is sequential organization . Integration involves relating what is 
taught in one part of the educational programme to what is taught in another part . Effective 
organization of learning experiences involves planning both for sequence and for integration . 



The fourth major task in education is to evaluate the effectiveness of the educational 
programme in actaining its objectives, through appraising the educational progress of the 
students. This requires evidence of changes in students 1 behaviour during the time they are 
taking part in the educational programme. It means appraisal early In the course as well as 
near the end . It involves evidence relating to all of the important objectives which will 
help u9 to identify those aspects of the curriculum that are effective and those that need * 
improving . 

With this brief review of the essential characteristics of a profession and the major 
tasks in planning and conducting an educational programme, we can identify more easily the 
distinctive attributes of education for the professions. 

Objectives of professional education 

The existence of principles and of a code of ethics in a profession points to several 
kinds of educational objectives which are important in educating for a profession. The 
development of an ethical practitioner who has an adequate understanding of the ethical code 
of the profession, who applies the ethical principles intelligently to the varied particular 
instances that arise, and who is sincerely committed to the highest ends of the profession, 
requires an education programme which consciously aims at several major types of objectives. 

In terms of knowledge and understanding, a programme of professional education needs 
to develop in students a broad and clear concept of the social role of his profession . This 
includes understanding of the social functions the profession serves and how these functions 
arc related to the total functioning of society and to the functions of other major specialized 
groups. It also includes an understanding of the various kinds of relations which exist 
between the profession and society at large and between the profession and other specialized 
groups, including the expectations which these groups hold regarding the members of his profes- 
s ion . 
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Also, in terms of knowledge and understanding, professional education aims at developing 
a deep understanding of the persons for whom the professional service is rendered including 

particularly insight into personal motivations, reelings, needs, and the interrelation of 
physical,, psychological, social, and emotional aspects of human behaviour. Furthermore, 
professional education needs to include among its objectives the development of self-understan - 
ding in the students v Obviously, in no sense car. complete self-understanding be attained by 
the time of graduation from the professional scho'l, but a s»-ficient beginning can have been 
achieved to provide for continued developjaeirtr -as he pursues his professional career. 

In terras of effective thinking or problem-solving, the objectives of professional 
education which are derived from the importance of .*tnics include the ability to recognize 
ethical problems, the ability to identify the ethical principles at issue , and the ability 
to work out appropriate courses of action in terras of ethical principles . 

In terms of attitudes, education for the professions aims at developing loyalty to the 
Asocial well-being of the persons who are x served by the profession, concern for a truly social 
role on the part of the profession, a sense of self-respect for che ^.oeial contributions of 
his profession and of his own work, anjt' a warm, cc«pcing, yet objective .attitude towards his 
clients. This involves developing in professional students a considerable degree of emotional 
maturity so that they are free to express and rece+ve emotionally charged communication and at 
the same tiime can act intelligently as new problems arise . 

3 The foregoing objectives clearly represent a l*rge task for piofessional education, but 
these are not all the important objectives. The requirement that the professional prac- 
titioner operate on the basis of principles rather than rules implies certain additional 
objectives if the educational programme is to aim consciously to develop students who can 
operate in this fashion. Some of the purposes outlined above contribute to the performance 
of professional duties in terms of principles . In addition, in terms of knowledge and under- 
standing there is need to understand the structure and functioning of the institution in which 
the profession operates , whether this be school, church, hospital, court, or other institution. 
Only with such a broad view can a professional practitioner operate with intelligence and on 
the basis of principles adequate to produce* desired consequences. 

Finally, of course, every professional school needs to aim consciously to develop an 
understanding of those principles, concepts, facts, and procedures which are basic to profes - 
sional operations . In medicine these include principles of physiology, anatomy, chemistry, 
physics , /bacteriology, and psychology. The tendency, however, is to limit these basic 
, principles too narrowly. In a v*ry real sense, doctors deal with^ problems in their normal 
professional work which are ps>ww:>logical and require an understanding of relevant psycho- 
logical principles if they are to operate intelligently. 

In terms of effective thinking, of problem-solving , it is clear that the foregoing 
objectives involving knowledge imply the development of some skill in recognizing professional 
p roblems,, in analyzing the problems in terms of the relevant principles, ^ nd in working out 
courses of action by applying these principle s. 

In terms of attitudes, the use of principles, rather than r^les in a profession, requires 
as objectives in professional education the development of broa^, rather than narrow, interests 
in the fields on which the profession draws, and the development of the student's interest in 
continuing his own learning long after graduation f*om a professional school . 

✓ 

Planning learning experiences 

The previous section outlines characteristic objectives of professional education. What 
about the learning experiences which the profession^ may use to attain these objectives? 

The most common problems in professional schools in connexion with the learning 
experiences used are: (1) the failure to select learning experiences in terms of the objec - 
tives to be attained ; (2) the failure to utilize consciously appropriate learning procedures 
for developing problem-solving skills, attitudes, and interests ; and (3) the failure to 
develop effective motivation for learning . These arc serious deficiencies. 
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Since learning is an active process , in which the learner himself is definitively involved, 
motivation is essential . Since the learner learns more than knowledge of content, and he 
actually learns what he is doing, what he is feeling, what he is thinking, , it is important to 
make conscious plans fo^students to learn to solve problems,, to develop attitudes and 
interests. Since objectives represent the aims of the professional school and learning 
experiences are/tfhe means of reaching educational aims, the latter should be planned in terms 
of these aims. 

An essential factor in planning learning experiences in terms of the objectives sought is 
to have clearly in mind what the objectives imply, both as regards behaviour to be developed 
and content involved. For example, if the objective "understanding the social functions the 
profession serves and how these functions are related to the total functioning of society and 
to the functions of other major specialized groups" is to serve as a guide for planningt 
learning, we must have a clear idea of what is "meant by "understanding " as a type of behaviour 
.to be developed and what content is included in the phrase beginning "the social functions the^ 
profession serves". $ost instructors who have sought to def jpe-*^Jn^et«tanding tt indicate that 
it is a mental process that is more active than memorization, since it involves not only 
remembering but also the ability to explain the concept or principle in one's own words, the 
ability to interpret , to illustrate , and to compare and contrast it to related ideas. Such a 
definition clarifies the behaviour that the student is expected to develop and, as pointed out . 
earlier, ?t suggests the kind of learning experiences that are required to attain the objec- 
tive. These would be learning experiences which give the student opportuni ty to explain, to 
interpret, to illustrate, and to compare and contrast it to related ideas . By defining the 
content implied by the phrase beginning "the social . functions the profession serves 51 , we are 
able to. identify what the principles and concepts are which the students will be given 
opportunity to explain, to interpret, to illustrate and to compare and contrast with related^ 
ideas. 

Correspondingly, as we define each objective in terms of the behaviour and content 
implied, it is a much easier step to select learning experiences that give students a chance 
to practice the behaviour involved and to utilize the relevant content. In this way, learning 
experiences are planned in terms of the objectives sought. 

The fairly common practice in professional schools of giving almost exclusive attention 
to knowledge and to technical skills may be partly due to the failure to recognize that 
effective thinking, attitudes and interests are also learned and can be consciously developed 1 
in programmes of professional education. Skills in ways of thinking are acquired by practice 
in solving problems. A new attitude is acquired by repeatedly observing* and reacting to 
certain phenomena from a new perspective. Interests in certain kinds of activit ies are 
developed as the student gains satisfaction from participating in these activities . For each 
,of these types of objectives, definite learning experiences can be provided so that students 
are practicing problem-solving, they are injected into situations from a new perspective, they 
they are able to gain satisfaction from certain kinds of activities. In short, professional 
schools can plan learning experiences for developing effective thinking, attitudes and 
interests. m . 

This leads to consideration of the primary significance of motivation. Since the learner 
learns through his reactions, unless he can be involved in the situation", unless he can be 
guided to think, feel, and act in ways appropriate to the situation, it is not possible for 
hi.m to learn. Practice alone, even when carried to unusual limits, does no t take the place 
of the "learner's being involved in what he is doing . 

Organizing learning experiences 

There are not only problems involved in the selection of learning experiences but also in 
their organization. As the programmes of professional education have become more complex and 
involved larger staffs, they have become more disintegrated. Yet significant changes in 
behaviour, that is, fundamental learnings, require a long time to develop. It is necessary 
that what is learned this term builds upon what was learned last terra, that what will be 
learned next year builds upon what is learned this year. This is sequence learning. 
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However, effective sequence is not simply a series of repetitions from one year to 

another* Rather, it includes variety in the learning experiences, so that each subsequent 
term emphasizes the main things to be learned, but in varied contexts . In this way, ever 
broader and deeper learnings are achieved • * 

Furthermore, effective organization provides for relations from one course to another and 
from one field to another which reinforce the learning in each course or field. This is done 
both by helping the student to use things learned in one course or field ir» another , and by 
helping him to perceive differences as well as similarities in the concepts, principles, atti - 
tudes and skills utilized in the various courses and fields * This is called curriculum 
integration . 

Sequence and integration are essential to programmes of professional education, but 
because of the tendency towards specialization and separation, conscicus efforts are required 
to plan for and develop effective organization. 

One significant development in working out a better organized curriculum is the building 
of a closer and more appropriate connextion between theory and practice, between the art and 
the science of the profession. Many institutions are not only teaching general principles 
but are also helping student^ to apply these principles to particular cases so that it is 
possible for them in practice to use the principles as they deal with particulars . Effective 
professional education requires this close connexion between theory and practice. Without 
theory, practice becomes chaotic, merely a collection of isolated, individual cases. Theory 
gives meaning and unity to what would otherwise be specific and isolated cases. 

On the other hand, without practice theory becomes mere speculation. The realities of 
practice provide a check upon pure speculation,-^ test of the adequacy of theory; and also, 
practice provides the problems «hich must be dealt with by any comprehensive theory. Hence 
these efforts to connect theory and practice more closely are important contributions to 
professional education . 

Another illustration of this development is the increasing use of the case method . The 
case method involves the student in the study of a concrete and particular case. However, 
for him to understand this case and deal with it effectively, he must bring to bear the theory, 
the concepts, the principles, that are basic to the issues raised by the case. The 
internship provides concrete experiences which are interpreted in terms of basic theory in 
the accompanying seminars. This shuttling back and forth between general and specific aspects 
of" a profession helps to build an increasingly adequate context of concepts and principles by 
which a member of the profession can understand the operations to be performed in relation to 
the values to be attained. An adequate theory helps him to relate particular activities in 
an individual case to the larger social issues, to see the connexion between the activities of 
the daily operations of the profession and the welfare of society generally. 

However, ~for these types of educational programmes to be effective, more than superficial 
experience and explanation must be provided. The student, needs many opportunities to deal 
with situations in the basis of a careful analysis of them, to identify the values and 
principles involved, and through practice to develop artistry in devising means to deal with 
the situation in order to preserve these values . Ttus is the aim of efforts to knit theory 
and practice more closely together. 

Another effort at extending the sequential organization of professional education is the 
working out of definite plans for continuing education after the member of the profession has 
completed pre-service training and has been inducted into the initial activities of his work . 
Fifty years ago me4ical schools felt that their work had been done when their graduates had 
been admitted to" initial professional activity. As the years went by it became increasingly 
clear that many members of the profession did not grow after they began their work and that in 
some cases the older practitioner was much less competent than those currently beginning their 
profession. Few doctors made any effort to continue their education after graduation. 
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Now there is a marked tendency for professional schools to develop programmes of con- 
tinuing education, in some cases bringing the work of the, school to the practitioner in the 
field, in other cases setting up short courses, institutes, ">r long-term seminars for 
practitioners to take on the campus. 

Evaluating effectiveness of educational programmes 

The final attributes of professional education are those involved in the task of 
evaluating the effectiveness of the educational programme in att aining its objectives, through 
appraising the educational progress of the students . Four important attributes are too often 
neglected in current educational programmes. 

The first of these is conducting an appraisal in terms of all of the important educational 
objectives of the professional school . The common practice is to appraise the knowledge of 
the students and certain of their technical skills. In addition, many professional schools 
appraise the student's ability to solve problems as^xhese are presented in verbal form. 
Few institutions provide for careful, systematic appraisal of problem-solving in the profes- 
sional situation, and appraisal of professional interests and attitudes. Hence the school 
does not have a comprehensive picture of the achievements of its students in terms of its own 
purposes. ■ 

A second attribute, often neglected, may partly account for the shortcomings in the first. 
A comprehensive programme of evaluation uses varied devices for obtaining evidence regarding 
the educational progress of students. These devices include not only written tests and 
examinations, useful as they are, but also observations interviews , questionnaires, reports 
from the field, samples of the student's work; in short, any device which gives valid evrdence 
reg\rding the significant behaviour of the student. Few professional schools consistently 
evaluit<r their effectiveness* in such varied ways. They tend to limit their appraisal to 
written examinations and to rating forms. This does not provide adequate means for compre- 
hensive evaluation. 

Apprarsal of the progress of students toward the objectives of professional education 
requires evaluation at several points in his career. This is a third essential attrrbute. 
To get evrdence of progress requires at least three appraisals, one early in his attendance 
at the professional school, one near his graduation , and one after several yegrs of servrce 
in the profession . Some schools attempt annual appraisals during the student's enrolment. 
The changes made while in the school throw light on the immediate effectiveness of the school s 
educational programme, while the appraisal after the student has spent several years rn the 
profession provides evidence of the permanence of learning and the extent to whrch rt has 
achieved some continuity with professional experience. Unfortunately these studies of 
progress are quite rare. This may be partly due to the lack of appreciation of the way rn 
which systematic sampling of students and alumni small enough to permit the use of rndivrdual 
interviews, yet at the same time sufficiently representative to permit valid generalrzations 
regarding the populations from which the samples were drawn. 

The last attribute to mentioned here is the use of evaluation in improving the 
edu catronal programme as well as in providing information to g uide work with individual 
students . Too often, what little appraisal of student achievement is conducted results only 
in grades for the students. Actually an evaluation programme can serve as a helpful means 
for continued improvement and development of the professional school. The results of 
appraisal indicate the respects in which students are mak' g substantial progress and the 
respects in which expected development is not taking place. These suggest aspects of the 
educational programme whrch need re-thinking and re-planning to provide for improvement. 
Fur-hermore, as revisions are made in the programme, subsequent evaluation mdicates the 
relative effectiveness of these revisions Hence, appraisal provides a sound basrs for 
planning . 
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The .evaluation data also indicate the progress made by individual students and bring to 
attention both their strength and their difficulties. This information thus provides a 
sound basis for the guidance of individuals and gives a more substantial foundation for the 
individual student's continued planning of his own education* 

In summary, the distinctive characteristics of a profession, namely its ethical code and 
.its operating basis on principles, suggest the distinctive attributes of education for the 
profession. From these characteristics, important educational objectives can be derived. 
Because these objectives are complex and involve understanding, problem-sqiving, attitudes 
and skills, they require clear definition in order to develop effective methods for "their 
attainment. The difficulty of attaining the goals of professional education makes motivation 
of prime importance and effective organization of learning experience a necessity. Finally, 
a comprehensive evaluation programme is required to guide the continued development and 
improvement of the educational programme and to provide information for constructive work 
with individual students. The building of an effective programme of education for a 
profession is not easy but when attacked intelligently, systematically, and enthu siastically, 
it can be done . 

h 
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\ . THE AUTHOR OF THE PRECEDING PAGES* IS 

RALPH W. TYLER 

HE OBTAINED HIS DOCTOR'S DEGREE FROM THE 

UNIVERSITY OF CHICAGO IN 1927 

HALF A CENTURY AGO J 



HE WROTE THESE LINES IN 1951 

A QUARTER OF A CENTURY AGO J 

IN SPITE OF THIS THEY REMAIN A REALITY OF TODAY. 

* 

FOR THOSE INVOLVED IN CURRICULUM- PLANNING THIS CLEARLY MEANS 
THAT PATIENCE AND PERSEVERANCE IS THE ORDER OF THE DAY. 



* The paper was presented on January 30 ,1952, In New York at the Annual Meeting of the 
American Association of Schools of Social Work and printed in the Social Work Journal, 
April 1952. 
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'EXERCISE 




(Check your answers on page 273) 

Instructions = for each question select ONE answer. 

\ 

Teachers frequently complain of the lack of motivation of students in 
explaining an unexpected number ojE failures. Which of the following concepts 
is reflected by this attitude: \ 

\ 

The primary role of the teacher is: \ 
A. To facilitate learning by the students. 





Question 8 . 



B. To transmit knowledge to the studentsX^ 

C. To guide the cognitive development of the students. 

* \ 

\ 

D. To encourage increase of responsibility o£ the students for their own 
learning. * \ 

E. To promote the motivation of the students, 
(true or false type) 

According to ^present educational trends 

2. The systematic approach helps to formulate hypotheses regarding the 
process of acquisition of competencies and facilitates the choice of 
relevant learning situations (true-fslse) . 

3. The primary role of the teacher is to transmit knowledge within the 
field of his specialty (true-false). 

4. The teacher helps the student' to learn ^nd the converse is also true 
(true- false) . 

5. Modern educational trends lay more stress on the teacher than on the 
student (true- false) . 

6. The student's judgement of the value of an educational activity is 
perhaps more relevant than that of the examiners (true-false). 

7. Modern educational trends transfer part of the role of the teacher to 
the student (true- false) . 

Indicate which of the following factors seems to be least likely to bring 
about a transfer of knowledge: 

A. Teaching is modelled on the organization of each discipline. 

B. The teaching environment resembles the professional practice environment. 

C. Teaching calls for active student participation. 

D. Teaching is centred on the health needs of the population. 

E. Teaching aims at developing compiex cognitive abilities. 
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Question 10. 



Question 11 . 



The many definitions of learning all include the following characteristics, 
EXCEPT ONE. State which: 

A. A more or less permanent change occurs in the student f s behaviour. 

B. It is not directly observable. 

C. Repetition by the student is a favourable factor. 

D. The role of the teacher is primordial. 

E. Motivation rs an essential element. 

Research has shown all the following factors, cept one , to be effective in 
maintaining motivation in the student: 

A. Presence of gratifications suitable for ensuring a continuity of learning^ 
Presence of a punishment system suitable for controlling deviant behaviour 



B. 
C. 



Detailed description of the objectives pursued and communication of the.se 
objectives to the students. 

D. Presence o„f an evaluation system frequently measuring the motivation level 
of the .STudents. * 

E. Presence of educational activities suitable for stimulating active 
learning. 

When preparing part of the curriculum whose aim is to make students capable of 
showing creativity in looking for solutions to medical problems, the most 
suitable learning activities for that objective are: 

A. The students participate in courses and demonstrations by teachers with 
a very creative attitude who are themselves excellent research workers. 

B. The stu !ents participate, in laboratory and field exercises during which * 
they reproduce basic and classic experiments. 

C. The students participate in a planned series of hospital, field and 
laboratory activities, and must present the solutions to exercises in 
clear terms. 

D. The students participate in hopsital, field and laboratory activities and 
must write a report describing the experiments in which they have partici- 
pated* 



Question 12 . A, curriculum specification table: 

Is a tabl^e giving details of a curriculum (time-table, premises, etc.) 

Is a contlol sheet for checking whether the elements of the curriculum 
are related to the educational objectives. 



A. 
B. 



C. Is a control sheet for checking the specificity of the educational 
objectives . 

D. Is a list of contents giving detailed instructions concerning programmed 
learning. 
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Question 13, 



Which of the following criteria (for selection of educational material) does 
not agree with any of those defined by Penta & Telder: 

The material should: 

A. be appropriate to educational objectives; , 

B. promote active student involvement; 

C. be appropriate* to the students; 

D. be of good technical quality; 

E. be very cheap. 




Question 14 . 
Question 15 . 



Question 16 . 
Question 17. 



Indicate for each of the following questions whether reference is made to 

A. an epidiascope 

B. a diascope 

C. a slide projector 

Can project outlined figures (or letters) on a transparent support (25 
so that they can be read in a lighted room. 

Can project outlined figures (or letters) on opaque paper so thatf they can be 
read in a darkened room. 

Is also called an "overhead projector 11 . 

Can project outlined figures (or letters) on a transparent support (50 x 50 mm 
24 x 36 mm) so that they can be read in a darkened room. 



Questions 18 to 20 . 



(true or false type) 



According to L. Weed: 

18. Length of training should be fixed in advance for a given group of students 
depending on the type of degree (true- false) . 

19. The student should be able to define his own work objectives within an 
organized system of medical care (true-false). 

20. Teaching of basic principles should precede the introduction of students to 
practical problems (true- false) . 
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Answers suggested for the exercise on pages 267 to, 269 



QUESTION 


SUGGESTED ANSWER 


If you did not find the correct answer, consult 
the following pages again 


1 


B 


205-206 


2 


True 






3 


Falue 






4 


True 






5 


False 




Y 205-206 

\ 


6 


True 






7 


True 


) 




8 ■ 


A 


114, 208, 210, 214 


9 


D 


213 
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CONGRATULATIONS | YOU HAVE WORKED OUT A 
CURRICULUM CORRESPONDING TO YOUR SUBJECT. 

BUT BE CAREFUL * * * 

- MAKE SURE ONCE MORE THAT IT AGREES WITH 
THE INSTITUTIONAL OBJECTIVES. 

- PAY CAREFUL ATTENTION TO RELEVANCE. 

v. 

- EFFECTIVE TEACHING CAN BE MORE 
DANGEROUS THAN NO TEACHING AT ALL IF 
IT IS NOT REALLY RELEVANT. 

* 
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EVALUATION 



This third chapter pre/eats basic concepts in the domain of educational evaluation. It 
stresses the very close relationship between evaluation -and definition of educational 
objectives; Td'the primary role or any evaluation, which is to facilitate decision-making by 
those responsible for an educational system. It defines the subject the purpose, the goals 
and the stages of evaluation. It shows that a classification of intellectual processes can 
be of considerable help in constructing an evaluation system. 

Those who would like to learn .nore concerning these problems should consult the following 
publications: 

^ Public Health Papers No. 52, WHO, "Development of educational programmes^ for 

the health professions", 1974 
^ Evaluation of school performance. Educational documentation and information. 

Bulletin of the International Bureau of Education No. 184, third quarter 1972, 

84 pages. 



After having studied this chapter ano the reference documents you should be able to: 
illustrate the relationship between evaluation and the other elements of the educational 
process ; 

define the major role of evaluation; 

determine the purpose of evaluation; 
^ mention six decisions which you feel you can take thanks to evaluation; 
(5^ describe four steps in the evaluation process; 

^ define the difference between formative and summative evaluation; 
J^, describe the general methodology of evaluation; 

seler the evaluation method most adapted to a given objective; 
^ define the following terms: validity, reliability, objectivity, and indicate what 

relationship exists between them; 

draw u ? an organizational diagramrepresenting an evaluation system for your establishment 
identify obstacles and relevant strategies to improve an existing evaluation system. 
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EVALUATION IS 




A DECISION ENABLING PROCESS 



FOR BETTER EDUCATIONAL DECISION MAKING 
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PLACE OF EVALUATION 
IN THE EDUCATIONAL CYCLE 
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NOTICE 



* * 

EVALUATION OF MEDICAL EDUCATION 

MUST BEGIN WITH A CLEAR AND 

MEANINGFUL DEFINITION OF ITS 
OBJECTIVES 
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Exercise 




evaluation! system 



Before starting to define the organization, stages or methods of an 
suitable for the establishment in which you are teaching, it would be useful to state: 

What important DECISIONS 1 you feel could be made on the basis of value judgements made 
possible by evaluation ? 

To decide .... 



1 Examples of "decision": 

# to decide what students will be aLlowed to move up from the second to the third year 

* or to decide to purchase an overhead projector rather than a blackboard 
or to decide to appoint Mr X Full professor. 



EVALUATION 

A FEW ASSUMPTIONS* 



* Education is a process, the chief goal of which is to 
bring about changes in human behaviour 

* The sort of behavioural changes that the school attempts 
to bring about constitute its objectives 

* Evaluation consists of finding 'out the extent to which 
each and every one of these objectives hat been attained, 
the quality of teaching techniques and teachers. 



* adapted from DOWNIE 
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Assumptions underlying basic educational measuremen t and evaluation' 

Human behaviour is so complex that it cannot be described or summarized in a single score. 

The manner in which an individual organizes his behaviour patterns is an important aspect 
to be appraised. Information gathered as a result of measurement or evaluation activities 
must be interpreted as a part of the whole. Interpretation of small bits of behaviour as 
they stand alone is of little real meaning. 

The techniques of measurement and evaluation are not limited to the usual paper-and- 
pencil tests. Any bit of valid evidence that helps a professor or counsellor in better 
understanding a student and that leads to helping the student to understand himself better is 
to be considered worthwhile. Attempts should be made to obtain all such evidence by any means 
that seems to work. 

The nature of the measurement and appraisal techniques used influences the type of 
learning that goes on in a classroom. If students are constantly evaluated on knowledge of 
subject-matter content, they will tend to study this alone. Professors will also concentrate 
their teaching efforts upon this. A wide range of evaluation activities covering various- 
objectives of a course will lead to varied learning and teaching experiences within £ course. 

The development of any evaluation programme is the responsibility of the professors, the 
school administrators, and the students. Maximum value can be derived from the participation 
of all concerned. 



* Downie, N. M. Fundamentals of Measurement: Techniques and Practices . New York: Oxford 
University Press, 1967. 
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A CONTINUOUS PROCESS « 
BASED UPON CRITERIA 
COOPERATIVELY DEVELOPED 
CONCERNED "MTH LEARNERS' BEHAVIOUR 



The philosophy of evaluation ' 

1. Each individual should receive that education that most fully allows him to develop his 

> 

potential. 

2. Each individual should be so placed that he contributes to society and receives personal 
satisfaction in so doing. 

3. Fullest development of the individual requires recognition of his essential individuality 

* 

along with some rational appraisal by himself and others. 

4. The judgements required in assessing an individuals potential are complex in their 
composition, difficult to make, and filled with error. 

5. Such error can be reduced but never eliminated. Hence any evaluation can never be 
considered final. 

6. Composite assessment by a group of individuals is much less likely to be in error than 
assessment made by a single person. - 

7. The efforts of a conscientious group of individuals to develop more reliable and valid 
appraisal methods lead to the clarification of the criteria for judgement and reduce the 
error and resulting wrongs. 

8. Every form of appraisal will have critics, which is a spur to change and improvement. 



* Downie, N. M. Fundamentals of Measurement: Techniques and Practices . New York: Oxford 
University Press, 1967. 
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The psychology of evaluation '' 



1 # For evaluation activities to be most effective, they should consist of the best possible 
techniques, used in accordance with what we know to be the best and most effective 
psychological principles* 

2. For many years readiness has been recognized as a very important prerequisite for 
learning. A student is ready when he understands and accepts the values and objectives 
involved* 

3. It has long been known that people tend to carry on those activities which have success 
associated with their results. This has been known as Thorndike's Law of Effect* 
Students in any classroom soon come to realize that certain types of behaviour are 
associated with success - in this case, high marks on a test or grades in a course. 
Thus, if a certain teacher uses tests that demand rote memory, the students will become 
memorizers. If a test, on the other hand, ; equires students to apply principles, 
interpret data, or solve problems, the students will study with the idea of becoming best 
fitted to do well on these types of test items. In the long run, the type of evaluation 
device used determines, to a great extent , the type of lear ning activity in which 
students .will engage in the classroom . 

4 # Early experiments in human learning showed that individuals learn better when they are 
"constantly" appraised in a meaningful manner as to how well they are doing. 

5. The motivation of students is one of the most important - and sometimes the most 
difficult to handle - of all problems related to evaluation. It is redundant for us to 
say that a person ! s performance on a test is directly related to his motivation. 
Research has shown that when, a student is really motivated, per formance is much closer to 
his real top performance than when motivation is lacking. 

6. Learning is most efficient when there is activity on the part of the learner. 



"Continuous 11 evaluation 



We know that evaluation of education should begin with a clear and meaningful definition 
of its objectives. We cannot measure something unless we have defined in advance what it is 
we wish to measure. 

When this phase of evaluation comprising the definition of objectives has been properly 
completed, the choice or development of suitable examination procedures is that much simpler. 
Schematically represented, the education cycle (page 304) comprises the determination of 
objectives, the development of teaching activities and the development of evaluation procedures 
for possible revision of objectives. Evaluation should be "continuous 11 otherwise its function 
will merely be one of penalization. It should not simply be a course of obstacles which the 
students are supposed to get over and which becomes their sole subject of concern, the actual 
instruction becoming quite secondary. Under these circumstances, the student's only interest 
is how to obtain his diploma at least cost. The teacher is entirely at fault if the student 
is not convinced that the purpose of his education is not merely to obtain a diploma . . . and 
that the purpose of evaluation is to help him obtain it. 



* Downie, N. M. Fundamentals of Measurement: Techniques and Practices . New York: Oxford 
University Press, 1967. 
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Teaching and performance level 



The aim of teaching is to bring about an appropriate modification of behaviour, in the 
field of attitudes and practical skill, ^based on indispensable knowledge. The objective is 
thus: what we want the student to be able to DO on completion of the instruction, which he was 
unable to do initially, keeping in mind the "minimum* level of performance 11 , and laying stress 
on the observable and measurable aspects of the objective. The minimum level of performance 
is the minimum that the student must do to convince the teacher that he should be allowed to 
continue his studies or that he should be allowed to begin practising his profession without 
assistance or supervision. 

Thus, in constructing an examination, it is essential to define the minimum level of 
performance (see also minimum pass level = MPL) that the student should achieve. 



ROLE OF TAXONOMIES IN EVALUATION (EXAMINATIONS) 3 



Classifications of educational objectives can, as we have seen, be used for curriculum 
planning and research. Taxonomies can also play an important role in designing achievement 
tests. , * 

1. Building logic into achievement tests , t 

Classification of achievement tests and test items is normally based on the content or 
form of the items. However the classification provided by the different authors is based on 
modifications in human behaviour which, from the educational standpoint, is more significant 
-and avoids many ambiguities. * 

A* sound evaluation system should be based on educational objectives. Logical 
-classification of these objectives provides a logical scheme of classification of evaluation 
material based on objectives. What matters is not that it should be perfect but simply that 
it exists. It is not possible to measure without a unit of measurement. 

2. Making evaluation more comprehensive 

Generally speaking, in educational evaluation the cognitive domain is that which is 
investigated most frequently. Testing of knowledge is the dominant feature since it is an 
area which seems easy to investigate. Once the structure of educational objectives has been 
understood, it should be possible to make the evaluation devices more comprehensive and multi- 
dimensional, and to devise instruments for collecting evidence about objectives in different 
domains and at a higher level. 

3 # Providing criteria for the evaluation of achievement tests 

In educational testing, it is first necessary to select appropriate techniques for 
collecting evidence about the student's achievement. For this purpose, the nature and level 
of the objective must be taken into account. The taxonomic classification can be used to 
evaluate the capacity of the technique to produce the desired evidence. 

4. Developing a new typology of test items 

The taxonomy is not in itself a typology of educational objectives; but it can be used 
for developing a useful typology of test items. At present, the typology of test items is 
based on the form of the item, such as multiple choice type, or the form of the answer, such 
as essay type, "objective' 1 type, etc. With the addition of a new dimension of objectives it 
is possible to develop a more meaningful classification. For example, an item concerning 
a given topic may be of the knowledge-based multiple choice type, or of the application-based 

1 Adapted from R. H. Dave, Op. Cit. , p. 160. 
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.short answer type or of the analysis-based essay type and so forth. This .two-dimensional 
classification of test material would make testing more meaningful and precise. It would also 
•ensure better communication among evaluators, which is one of the functions of a good taxonomy.. 

5. Facilitating the construction of relevant test items 

Test items are cften constructed without conscious effort to base them on predetermined 
objectives. Emphasis on form and subject matter in constructing a test item, and not on the 
-objective, is likely to result in information-ridden test material. The taxonomic 
classification reminds the item writer of the need to take into account this important aspect 
of an item. The content of an item is valid to the extent that it corresponds to the 
objective it is intended to test through a given unit of its content. 

6. Facilitating the' planning of tests and the interpretation of results 

In planning a test it is essential to decide what weight should be given to the different 
objectives, and planning is easier if the examiner has a clear understanding of their 
hierarchical classification. This classification can be used as an aid in determining the 
value and significance of the test. Similarly, in interpreting ^test results, account must be 
taken of the objectives and their relative standing in the hierarchical structure. Scores 
can be analysed according to the objectives, and the value of the learner's achievement can be 
appraised more meaningfully. This appraisal has a diagnostic value and may be used to improve 
the instruction. -Depending on the level of performance in respect of the different 
objectives, emphasis can be laid in the instructional programmes on those objectives which are 
more difficult to- attain. Thus taxonomic classification is useful in raising the educational 
value of achievement testing. , ,y 



RECOMMENDATIONS. 



While there are definite advantages in using taxonomic classification for constructing and 
improving achievement tests, they also involve difficulties. To start with, the construction 
of test items and other evaluation devices at higher levels in various domains is very 
exacting- it requires a mastery of the methodology of test development and creativity in the 
production of items. As the classification is hierarchical , the lower classes are included 
i n the next higher class, and this overlap is often an elem ent of confusion in test 
construction" These difficulties are particularly acute at the initial stages in the process 
of developing "objective" tests. However, practice and experience gradually give the 
examiner confidence and help him to.use taxonomies. 

Taxonomies are indispensable tools for teachers, enabling them to define educational 
objectives with due precision. 

For some teachers, painstaking development of detailed objectives for their courses 
becomes a sort of religion. Others ..ave heard about this approach and its potential value 
in improving the quality of their instruction, but have somehow, so to say, become completely 
immune to it since their first exposure to it. Finally, there are teachers, the great 
majority, whose many different tasks absorb the too short time available to them, but wno 
would like to be informed. However, more than this is required: they must be trained. 



* 
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Formative and \summative evaluation 



You will find the following equivalents in the literature for these two expressions: 



Formative evaluation 



diagnostic evaluation 



Summative- evaluation 



certifying evaluation 



1. ■ » Formative evaluation 

- aims at informing the student on the amount of learning he still has to make 
towards achieving educational objectives; 

- consists in evaluating the PROGRESS or the GAINS mad* by the student from the 
moment he commences a curriculum until the time he completes it; 

- enables the curriculum to be adjusted in accordance with the progress made, or 
its absence; 

- SHOULD IN NO WAY BE USED BY THE TEACHER TO MAKE A CERTIFYING JUDGEMENT; 

the anonymity of the student should be safeguarded (he can use a code of his own 
choice) ; 

- is very useful in assisting the student to ask for help and the teacher to 
provide help; 

- provides the teacher with qualitative and quantitative data for changing his 
teaching, or leaving it unchanged (including educational objectives). 

2. ' » Summative evaluation 

- serves traditionally for rank-ordering students and justifying decisions regarding 
their passage to the following year or the obtaining of a degree; 

- is effected at the end of a unit of instruction (or group of units). 



* Read the article by C. MacGuire - diagnostic examinations in medical education. 
Public Health Papers No. 52. 
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3 # This distinction between FORMATIVE evaluation and SUMMIT IV E evaluation can also apply to 

♦ 

curriculum evaluation. 

Scriven terms "formative" the evaluation employed to improve a curriculum during its 
implementation* The inadequacies and strong points of the curriculum are identified during 
its application and in son,e cases adjustment* can be made to remove certain weaknesses. 
Summative evaluation, on the other hand, is the final evaluation of the curriculum. It 
represents an effort to reach general conclusions of the type required to justi'fy a decision 
to replace one curriculum by another or to accept (reject) a given curriculum, book, teaching 
material, etc. Although formative evaluation is normally carried out by the personnel f 
participating in curriculum improvement whereas summative evaluation is more often undertaken 
by those who are not directly engaged therein, the distinction between the two concepts is not 
as clear-cut as might appear. It can be argued, for example, that fornu C-/* evaluation is 
only *• step in the direction of summative evaluation.*" 



1 In: Evaluation of school performance. IBE Bulletin No, 184 (1972). The reader is 
referred to this very instructive prblication. 
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^ EVALUATIO 



N 



1 

of WHOM* I 

7 

of W H A T I 



1. STUDENTS 



TEACHERS 



3. PROGRAMMES AND ^'JRSES 



in relation to WHAT 



In relation to EDUCATIONAL OBJECTIVES. 
They are 

the comnon denominator. 
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STUDENT EVALUATION : WHAT FOR 







INCENTIVE TO LEARN 



FEEDBACK TO STUDENT 



19 I Modification of LEARNING ACTIVITIES 



SELECTION OF STUDENTS 




SUCCESS OR FAILURE 



FEEDBACK TO TEACHER 



23 SCHOOL PUBLIC-RELATION 






24 PROTECTION OF SOCIETY 



Numbers found in left side column refer to exercise page 346 
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Aims of evaluation 



1. To differentiate between failure and success by the student. This was the conventional 
ro?,e of examinations (summative evaluation). 

2. To provide "feedback" for the student; to keep him constantly informed about the 



instructions he is receiving; to tell him what level he has reached; and to enable him, 
through the questions, to realize what parts of the instructions he has not understood 
(formative evaluation). 

3. To provide "feedback" for the teacher; to make him eventually aware that a ^fairly large 
group of students have not understood the instruction he sought to give them; and to 
enable him, when necessary, to correct his teaching so as to ensure that the matter 
seeks to communicate to the students is correctly understood (formative B^aluation). 



4. - The "reputation of the school" is something whose importance is not always evident in 

European schools, since their reputation is based not on an examination system but on 
a long-standing tradition. On the other hand, North American medical schools, for 
example, customarily make known the percentage of students who have passed "National Board 
of Medical Examiners" examinations (formative evaluation). 

5. Protection of society. It is the teacher's responsibility towards society. One should 
know the extent to which graduates constitute a potential danger (summative evaluation). 




Adapted from Downie, N. M. Fundamentals of Measurement. Techniques and Practices . 



New York: Oxford University Press, 1967. 
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STEPS IN EVALUATI 



1. Defining objectives 

including criteria (minimum level of performance) 



2. Obtaining and using measurement instruments 



3. Interpreting measurement data 



4. Making judgements and Taking appropriate action 
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EVALUATION 

• * * 

COMMDN METHODOLOGY FOR EVALUATION 

*of cognitive processes 
*of .attitudes 
*of skills 

•k 

1 # Make a list of observable behaviours showing that the objective pursued has been reached. 

2. Make a list of observable behaviours showing that the objective pursued has not been 
! reached* 

3. Determine the essential behaviours in both lists, 

4. Give positive or negative weighing to the components of these cwo "I^ts 

5. Define the minimal performance score. 

+ Obtain the agreement of several experts for the last three stages. 



Behaviours are expressed positively in the two lists. 

Example: Objective : sympathetic attitude towards a sick child. 

At the two extremities of a descriptive scale of values there could be: 

Cuddles the patient . . . Slaps the patient's face. 
See also Rezler, Public Health Papers No. 52, pp. 70-83. 
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LEARNING OBJECTIVES AND METHODS OF EVALUATION 



COGNITIVE 



DOMAINS 
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AFFECTIVE 


PSYCHOMOTOR 
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TESTS- 



"OBJECTIVE" 
"ESSAY TYPE' 
'SIMULATION 



t 



PRACTICAL TESTS 



DIRECT OBSERVATION 



INDIRECT 
METHODS 
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ORAL TESTS 
QUESTIONNAIRES 



— Complex cognitive elements 

— attitudes 

— skills 
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I Exercise : 

t 

For each of the 9 objectives you have already defined (pa^ 165), choose one of the 
methods of evaluation (on page 324) which you feel is best for informing you and informing 
* the student on the extent to which each objective has been achieved: 



Obj ecnive 


Method of evaluation 




1 page 149 


•Indirect method = completion type question 


e 

X 


2 page 149 


* 

Indirect method = questionnaire 


a 
m 
P 


3 page 149* 


Direct observation = practical examination 


1 

e 
s 


1 




2 




3 




4 




5 




6 




7 




8 

* 
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Attitude measurement 



The term "attitude" refers to certain regularities (consistencies) of an individual's 
feelings, thoughts and predispositions to act in response to an aspect of his environment 
(affective domain). An attitude is usually thou^nt of as a hypotheti cal construct, not 
directly open to observation but which can be inferred from verbal expression or overt 
behaviour. 

Attitudes are probably not innate. An individual's entire personality structure and 
hence all his behaviour may be thought of as organized around a central value system composed 
of many related attitudes. 

Contemporary research seems to show that it is unrealistic to hope to be able to change 
basic attitudes in the relatively short period of an educational programme. Consequently it 
would seem preferable to make sure, by means of suitable selection, that students have 
attitudes appropriate to their future profession on entry . We must, however, remember that 
the definition of these attitudes may readily lend itself to dispute, so that their prepara- 
tion must be of a collective nature and exclude any arbitrary .judgement. Moreover, it would 
be wise to carry out thorough, controlled experiments before taking any action on the basis of 
results obtained in at titude measurement . In the face of such a situation it is not unreason- 
able to consider the drawing of lots as a method of selection. 

The attitude of the physician and of other members of the health team towards the patient 
and his family is universally recognized as having very great importance. 

However, on considering the manner in which the performance of students in this field is 
measured, it is found to be so faulty that i.t is impossible not to be struck by such 
a paradoxical situation. 

It must be admitted, in all humility, that this is certainly the field where measurement 
is most difficult and we should support the efforts of those who are undertaking relevant 
research. There can be no question of dealing with this problem thoroughly within the limits 
of the present superficial study and the reader should consult specialized publications (see 
references) • 

Scales are utilized to evaluate reactions, attitudes, and student activities. Because 
of the ease with which they can be used, many sources of error in evaluation are often 
overlooked. 

If we wish, in observing a student who is examining a patient, to evaluate separately the 
extent to which the student gives the patient confidence , the following scale may be used, 
either in numerical form (on the right) or in graphical form (on the left): 

The student has taken all the necessary precautions and the patient 
" seems completely relaxea 5 



The student has taken the necessary precautions and nas on several 

occasions reinforced the patient 1 s confidence ♦ 4 

The student has made an effort in this direction and has noted the result . . 3 

The student has made an effort without attempting to determine the result . ♦ 2 

The student does not appear to have even envisaged the problem 1 
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These two examples show the scales most frequently used* For other scales see 
Rezler, A» D, The assessment of attitudes , Public Health Papers No* 52, pp. 70-83 and 
Guilford, J. P. Psychometric Methods , McGraw Hill, 1954. 

- In order to dr aw up questions for evaluating attitudes, a list of statements is prepared 
reflecting both attitudes deemed favourable and those deemed unfavourable in regard to the 
characteristic being measured . . - 




Draw up scales for evaluating attitudes expressed in educational objectives you 



prepared on page 165. 
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EVALUATION "BY 11 THE STUDENTS 1 



I. Introduction 



Student evaluation provides the t \acher with a feedback regarding the quality of his 
teaching. However, this feedback is of an overall nature and anyone who really wishes to 
improve his teaching will want to know more and must therefore request the student f s opinion . 
Such an approach may go no further than a friendly discussion with a few students, or can 
lead to the preparation and distribution of a questionnaire. In the latter case one may 
speak of an evaluation of the course BY the students. The evaluation questionnaire can deal 
with the whole or some special aspect of teaching: it is the teacher who decides what he 
wishes to have evaluated by the students. In the event, for example, of a change having been 
made in teaching (re-organization, utilization of audio-visual equipment) it is extremely 
useful for the teacher tc obtain the opinions of the students on that change. 

It may . be noted that anyone who asks students to evaluate his teaching need not doubt the 
validity of their judgement ♦ For a decade or sq many psychometric studies have revealed the 
validity and the _accuracy of .student opinions as well as their close correlation with 
"objective 11 measurements of the instructor's effectiveness. The many biases which were 
ascribed to the evaluators, for example, the influence of sex, academic efficiency, (poor 
students/ good students), level of studies (beginners, finishers) the status of the course 
(compulsory/optional) have all proved to be of negligible importance * 

II. Preparation of "simple" evaluation questionnaires 
• * 

Once the teaching aspects which are to be evaluated have been determined, the evaluation 
questionnaire is drawn up. 

\ 

This questionnaire must comply with several essential requirements as concerns the wording 
of the statements, the scale of the answers and the method of "administration". 

1) Wording of the statements 

(a) The statements should be clear, simple and directly related to what it is desired 
to measure; 

(b) statements which would be accepted straight away by everyone shculd not be included; 

(c) statements containing double negatives should be rejected; 

(d) each statement should contain a complete thought ; 

(e) the terms "uniquely", "solely", or "most of the time" should be avoided or used in 
moderation; 

(f) a statement should preferably be in the form of an affirmation concerning which the 
student is asked to express a favourable, neutral or unfavourable opinion; 

(g) the number of statements should be limited. Too long a questionnaire wearies the 
evaluators and this inevitably affects the validity of the answers. It is estimated 
that a questionnaire should not exceed 60 statements. 



* Summary of a paper presented by J.-F. d'lvernois, 21 May 1975, Seminaire de P6dagogie 
universitaire, Laval University, Quebec. 



Examples of statements 

"Specific objectives make it possible to identify what is most important in the 
content to be learned". 

"The time allocated for a lesson is sufficient for you to achieve all the objectives 
aimed at in that lesson". 

0 

2) Scale of answers 

The student should not have to answer merely "yes" or "no" to a statement, for such an 
answer provides only scanty or dubious information. 

That is why preference is given to the use of scales of answers comprising 5 degrees 
(and sometimes 7 or 9). 

Such a scale ranges from disagreement (1 and 2) through neutral (3) to agreement (4 and 5) 

Example : 

Disagreement Agreement 
Complete Partial Neutral Partial Complete 

1 r 1— 



Students should enter against each' statement (or on an answer-form or card) the numbers 
corresponding to their opinion. It is advisable to provide for an additional answer figure 
(0 to 9) corresponding to a lack of opinion concerning the statement . 

The answers scale should be explained to the students at the beginning of the question- 
naire, or mentioned against each statement. It is also as well to leave a space below each 
statement for comments by the student. 

3) Method of administration 

A person's reaction to a given stimulus is first immediate and then delayed. If the 
questionnaire is handed out immediately after the stimulus the results observed will 
different from those obtained if the questionnaire is administered later. These d fferences 
in distribution are attributable to the effect of experienced familiarity. The teacher 
should therefore take this factor into account when he envisages the administration of an 
evaluation questionnaire. 

4) Analysis and interpretation of the answers 

Processing (machine or manual) of the answers is simple The answers to a given state- 
ment are then analysed by calculating the number of students (frequency) who have replied by 
1 2 3 4 5 or 9 to that statement. The mean of the answers to the statement can also be 
calculated! To do this one point is assigned to an answer "1"; two points to an answer 2 ; 
three points to an answer "3"; etc. The. total of the number of points obtained for the 
statement is calculated and then divided by the number of students answering. To make inter- 
pretation easier, the trends (favourable, unfavourable) should be grouped together. 



* also called "practice effect 
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Example' t 

Statement 12 : i? The time allocated to a lesson is sufficient for you to achieve all the 

objectives aimed at in that lesson". 

/ • * » 

/ 

i 

Number of answers : 100 

Distribution : lM ' 3 3 : 5 5 : 69 
2:2 4 : 21 9:0 

Trends : 1-2 (disagreement) : 5 
4-5 (agreement) : 90 

<S # * j 

Mean of answers : 4.51 y v ^ 

i 

III. Construction of complex questionnaires 

1) "Simple" questionnaires enable the students 1 perception of REALITY to be measured. It 
is for the teacher who analyses the answers to deduce from them what the students desire or 
expect, but this interpretation may leave certain grey zones, i If, for example, the statement 
submitted is: "the course provides an adequate cover2ge of the'" subject matter" and 407*. of the 
students disagree with that statement then it is clear that these students feel that the^ course 
does not cover sufficient subject matter. On the other hand, what proportion of the 50% of " 
students who agree with the statement think that the subject matter covered is not only 
adequate but even too much? / 

"Complex" evaluation questionnaires, such as the one/devisea by F. Gagn6 r covering the 
perception J>y the. students of the teacher/students relationship have the aim of measuring - 
unambiguously as possible both the students 1 perception, of REALITY and their level of 
expectations. j 

In a questionnaire of the GagnS type, each statpnen' should .include: 

/ 

(a) a TITLE; / 

(b) a DETAILED DESCRIPTION of the aspect to be measured; 

(c) a 5 degree SCALE with QUALITATIVE EXPRESSION of each degree; 

(d) TV0 QUESTIONS : the first (A) measuring the students 1 perception of reality and 
the second (B) the level of his expectations. 



as 



FREQUENCY OF EXAMPLES : What is required to measure the frequency of examples putting 
in concrete form the more *• less abstract concepts presented in the course. 

1. VERY RARE examples > 

2. RARE exa* /les 

3. MORE OR LESS FREQUENT examples 

4. FREQUENT examples 

5. VERY FREQUENT examples 

Question A : Where would you place this course on the evaluation scale? 
Question B : Where should it be in order ^ satisfy you? 
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Analysis and interpretation 



The means of the answers to question A (Reality) and B (Expectations) are calculated 
for each statement. Interpretation o^these two means is easy since degree 1 on the scale 
shown usuallycorresponds to the minimum frequency or intensity, while degree 5 corresponds 
to maximum frequency or intensity (see example). The deviation between thfese two means 
is then calculated (S score). This deviation shows the dissatisfaction of the students, 
a dissatisfaction expressed in terms of a lack or an excess . The lower the value of the 

score (the nearer it is to zero) the more the particular aspect measured is deemed 
satisfactory. The S scores of several statements can be compared with one another for one 
and the same group of students. Furthermore, all the S scores in the questionnaire can be 
added together so as to give an overall and valid measurement of satisfaction or dissatis- 
faction. Finally, it should be noted that a questionnaire of the GagnS type can be adapted 
to different educational methods, e.g. lecture courses, /mall group activities, teaching by 
computer, by television, etc. A 



EXAMPLE OF INTERPRETATION OF A QUESTIONNAIRE OF THE GAGNE TYPE 



Statement 


Title of statement 


Mean : 
Reality 


Mean : 
Desires 


Deviation 
(S score) 


Interpretation 


4 


Variation in 
educational approach 


2.39 


3.74 


1-.52 


Dissatisfaction 
.Clack) \ 

! 


13 


Active student 
participation (+ or -) 


4.26 


4.65 


^0/39 


Satisfaction 


27 


Number of references 
to be consulted 


4.51 


3.27 


1.24 


Dissatisfaction 
(excess) ^ 

T 



See references concerning this article on page 515 et seq . 
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Now 

'construct a questionnaire that would enable participants to your next course to 



express their opinion concerning its quality. 
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EVALUATION IN EDUCATION 
THE CONCEPTS OF VALIDITY AND RELIABILITY 




1. Some definitions 

1.1 It should be recalled once more that education ib defined as a process developed for 
bringing about changes in the student's behaviour. At the end of a given learning period 
there should be a greater probability that behaviours regarded as desirable will appear; 
other behaviours regarded as undesirable should disappear, 

1.2 The educational Objectives constitute the desire d behaviours as a whole for which the 
teacher should prepare a suitable environment where the student can acquire them. 

1.3 Evaluation in education is a systematic process which enables the teacher to "measure 11 
to what extent the student has attained the educational objective . The evaluation always 
includes measurements (quantitative or qualitative) plus a value judgement. 

1.4 To make measurements, measuring instruments must be available which satisfy certain 
requirements so that the results can signify something to the teacher himself , the school, the 
student, and society which, in the final analysis, has set up the educational structure. 

1.5 In education, the measuring instruments are generally referred to as "tests". 

1.6 Among the requisite qualities of a test, whatever its nature, four are essential, 
namely, validity , reliability , o bjectivity and relevance . Others are also important, but 
they contribute in some degree to the qualities of validity and reliability. 



2 . Qualities of a measuring instrument 

The four main qualities of any measuring instrument (examination) are: validity, 
reliability, objectivity and relevance. 

2.1 Validity : the degree of precision with which the test employed really measures what it 
is intended to measure. There should be no outside factors to interfere with the manner in 
which the evaluation is carried out. For instance, in measuring the ability to synthesize, 
other factors, such as style should not compete with the element to be measured so that what 
is finally measured is style rather than the ability to synthesize. 

It is a concept which relates to the results obtained with a test and not to the test 
itself. It relates more specifirally to the interpretation of the results obtained by means 
of the test. 

The concept of validity is a very relative one. It implies a concept of degree, i.e. 
one may speak of very valid , moderately valid or not very valid results. 

The concept of validity is one which is always specific for a particular subject . For 
example, results of a test of public health administration may be of very high validity for 
identification of the needs of the country and of little validity for a coot/benefit or 
cost/e.Jficiency analysis. 

Content validity answers the following question: will this test measure, or has it 
measured, the matter and the behaviour that it is desired to measure? 

Predictive validity answers the following questions when it is desired to use the results 
of a test for predicting the performance of a student in another domain or in another 
situation: 

To what extent do the results obtained in physiology help to predict performance in 
pathology? 
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To what extent do the results obtained during the pre-clinical years help in predicting 
the success of students during the clinical years? 

2.2 Reliability : this is the consistency with which an instrument measures a given variable. 

Reliability is always connected with a particular type of consistency: the consistency 
of the results in time, consistency of results according to the questions; consistency of the 
results -rding to the examiners. 

Reliability is a necessary, but not sufficient, condition for validity. In other words, 
validity results must necessarily be reliable, but reliable results are not necessarily valid. 
Consequently, results which are not very reliable affect th<* degree of validity. Unlike 
validity, reliability is a strictly statistical concept and is expressed by me^ns of a 
reliability coefficient or through the standard error of tne measurements made 

Reliability can therefore be defined as the degree of confidence which can be placed in 
the results of an examination. It is the consistency with which a test gives the results 
expected. 

2.3 Objectivity : this is the degree of concordance between the judgements of indep. ident and 
competent examiners as to what constitutes a "good" answer for each of the items of a lu^asuring 
instrument. 

2.4 Relevance : this is the degree to which the criteria established for selecting questions 
(items) conform*to the aims of the measuring instrument are respected. This notion is almost 
identical to the one of content validity. In any case, both these qualities are established 
in a similar manner. 

3. Other qualities of a measuring instrument 

3.1 Equilibrium: the degree of concordance between the proportion of questions allocated t. 
each of the objectives and the ideal proportion for an instrument of this kind. 

Equity : "^degree of concordance between the questions set in the examination and the 
teaching content. 

Specificity : quality of a measuring instrument whereby an intelligent student who has 
not followed the teaching on the basis of which the instrument has been constructed will obtain 
a result equivalent to that expected by pure chance. 

Discrimination : quality of each element of a measuring instrument which makes it possible 
to distinguish between good and poor students in relation to a given variable. 

Efficiency : quality of a measuring instrument which ensures the greatest possible number 
of independent answers per unit of time. 

Time: it is wellknown that a measuring instrument will be less reliable if it leads to 
the introduction of non^elevant factors (guessing, taking risks or chances etc.) because the 
time allowed is too short. 

Length: the reliability of a measuring instrument can be increased almost indefinitely 
(Spearman-Brown formula) by the addition of new questions equivalent to those constituting the 
original instrument. 
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VALIDITY 



The degree of accuracy with which the instrument 
measures what it is constructed to measure 



RELIABILITY 



The consistency with which an instrument measures 
a given variable. 



OBJECTIVITY 



The degree of concordance between the judgements 
of independent and competent examiners as to what 
constitutes a good answer to each of the elements 
of a measuring instrument. 

* 



RELEVANCE 



Extent to which the criteria established for selec- 
ting questions so that they conform to the aims of 
the measuring instrument are respected. 
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RELATIONS between CHARACTERISTICS of an EXAMINATION' 



RELIABILITY 



OBJECTIVITY 



DIFFICULTY 



VALIDITY 




DISCRIMINATION 




HOMOGENEITY 

(exam.) 



HETEROGENEITY 
(students) 



RELEVANCE 



EQUILIBRIUM 



EQUITY 



SPECIFICITY 



EFFICIENCY 



DURATION 
(Time) 



LENGTH 
(number of Q) 



*as proposed by G. CORMIER Universite Laval, Quebec. 
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RELATIONSHIPS BETWEEN THE CHARACTERISTICS OF AN EXAMINATION 



The diagram on the previous page, suggested by G. Cormier, represents an attempt to sum up 
the concepts of doc imo logy worked out by a number of authors. However, no diagram can give a 
perfect representation of reality and the purpose of the following linearis to explain, rather 
than defend, the diagram. 

A very good treatment of all these concepts will be found in the book by Robert Ebel 
entitled Measuring Educational Achievement (Prentice Hall, 1965) 

Validicy and reliability 

Ebel shows that, to be valid, a measuring instrument (test) must be both relevanc and 
reliable. This assertion justifies the nitial dichotomy of tiie diagram. It is, moreover, 
generally agreed that a test can often, .f not always, be made more valid if its reliability 
is increased. 

Validity and relevance 

According to Ebel 1 s comments, it seems that the concept of relevance corresponds more or 
less to that of validity of content. In any case, loth are established in a similar manner. 

By definition, a question is relevant if it adds to the validity of the instrument, and 
an instrument is relevant if it respects the specifications (objectives and taxonomic levels) 
established during its preparation. 

Relevance and equilibrium 

It seems, moreover, that the concept of equilibrium is only a sub-category of the concept 
of relevance and that is why the diagram shows it as such. 

Relevance and equity 

It seems evident that if the instrument is constructed on the basis of a content -tself 
determined by objectives, then it will be relevant by definition. If this is not drne, then 
the instrument will not be relevant and consequently not valid. It is equitable the first 
case and non-equitable in the second. However, an examination can be equitable without being 
relevant (or valid) when, although it corresponds well to the teaching content, the latter is 
not adequately derived from the objectives. 

Equity, specificity and reliability 

The diagram reflects the following implicit relationship: a test cannot be equitable if 
it is not first specific. Moreover, specificity, just like equity and for similar reasons, 
will affect the reliability of the results. 

Reliability, discrimination, length, homogeneity (of questions) and heterogeneity (of 
students) 

According to Ebel reliability is influenced by the extent to which the questions (items) 
clearly distinguish competent from incompetent students, the number of items, the similarity of 
the items as regards their power to measure a given skill and the extent to which students are 
dissimilar with respect to that skill. The discriminating power of a question is directly 
influenced (see pages 469-470) by its level oZ DIFFICULTY. The mean discrimination index of 
an instrument will also be affected by the HOMOGENEITY of the questions and the HETEROGENEITY 
of the students. From the comments made above it can be seen how EQUITY and SPECIFICITY will 
also influence the discriminating power of the instrument. 
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ORGANIZATION DIAGRAM SHOWING RELATIONSHIPS BETWEEN 
CURRICULUM COMMITTEE, EVALUATION COMMITTEE AND 

TEACHING UNITS 



SCHOOL ADMINISTRATION 
TRANSMITS INSTITUTIONAL 
OBJECTIVES 



^ 



DUCATIONAL CONSULTANT 




CURRICULUM COMMITTEE 
PREPARES SPECIFICATION 
TABLES 



EVALUATION COMMITTEE 
PREPARES COMPREHENSIVE 
TEST 



The first year coordi- 
nator transmits the 
intermediate objectives 



The second year coordi- 
nator transmits the 
intermediate objectives 



I 

The Nth year coordi- 
nator transmits the 
intermediate objectives 



/ 



n 



\ 



\ 



EACH UNIT PREPARES THE SPECIFIC OBJECTIVES AND THE TEST ITEMS 

V V V V V V V i_l 

167 




Show-graphically the type of organization (commissions, committees, administrations, 
etc . 9 with a description of their functions ) which you feel desirable (in the establishment 
where you are teaching) for introducing (or improving) an EVALUATION system capable of 
providing the necessary data for checking that all INSTITUTIONAL objectives have beep achieved 
(do not forget also to take account of the decisions you have formulated on page 307); then 
compare with page 340. 
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Describe the obstacles you are liable to encounter in applying the organizational plan 
you have imagined on the previous page AND indicate tactics for overcoming each of these 
obstacles. 
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EXERCISE^ (check your answers against page 348) 



Inst ructions : For each question select ONE answer 



Question 1 



Question 2 



Question 3 



All the following steps, EXCEPT ONE, are essential in constructing any 
measuring instrument. 

A. Precisely defining all the aspects of the type of competence to be 
measured. 

B. Obtaining reliability and validity indices for the proposed instrument. 

C. Making sure that the type of instrument selected corresponds to the 
type of competence tc be measured. 

D. Making sure that tha measuring instrument is objective. 

E. Determining the particular behaviour expected from individuals who have, 
or ha/e not acquired, the specified competence. 

A test set at the beginning of a learning period (formative pre-test) has 
ail the following advantages EXCEPT ONE. It makes it possible to: 

A. modify the educational objectives of that period. 

B. devise mechanisms enabling less well prepared students to catch up. 

C. change the required pass mark. 

D. serve as a basis of comparison for measuring the real progress made. 

E. exclude inadequate students from the learning period. 

The main role of evaluation is: 



Question 4 



Question 5 



The purpose of evaluation is to make a valuable judgment concerning: 

A. Students and curricula. 

B. Students and teachers. 

C. Curricula and teachers. 

D. Students. 

E. None of the above statements. 

The four steps of the Evaluation process are the following: 

1 

2 

3 

4 
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Questions 6 to 10 - For each question indicate to which of the following it refers to: 



A. Formative evaluation. 

B. Summative evaluation; 

C. Both. 

D. Neither. 



Question 6 
Question 7 
Question 8 
Question 9 
Question 10 



Question 11 




Its main aim is to inform student on his her progress. 
Anonymity is not preserved. 

Enables the teacher to replace a programme by another. 
Enables to justify if a student passes or fails. 
Enables rank-ordering of students. 

Thorndike's "Law of Effect" is based on the fact that: 

A. Students learn better when they are motivated. 

B. Students learn better when they play an active role. 

C. Students are receptive when they understand the educational 
objectives which have been defined. 

D. Students tend co carry on chose studies which have success 
associated with their results. 

E. Students work better if the teacher makes an impression on them. 

and 1 3 J As regards the following qualities that can be attributed to 
an examination; 



A. = Validity 
D. -= Specificity 



B. = Objectivity 
E. - Relevance 



Re liahi lity 



12. 



What quality is obtained if a group of experts agree on what constitutes 
good answers to a test? 



13. What quality implies that a test consistently measures the same thing? 



Question 14 



The following factors, EXCEPT ONE, generally affect the reliability of 
a test: 

A. The objectivity of a test. 

B. The mean discrimination index of the test questions. 

C. The homogeneity of the test. 

D. The relevance of the test questions. 

E. The number of questions in the test. 
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Questl „„ IS ] Which OC t he foUo.ins «-« «i«U U «l— by .11 *• oth„„ 



A. 


Reliability. 


B, 


Validity. 


C. 


Objectivity. 


D. 


Specificity. 


E. 


Relevance. 



Question 16 



The most essential variable to be considered in evaluating the results of, 
teaching is: 

A. The student's performance. 

B. The opinion of the teacher and his colleagues. 

C. The opinion of the student regarding his performance, 
n The satisfaction of the teacher and the students. 



Question 25 



E. The teacher's performance. 

Questions 17 to 24 1 For each of the aims of student evaluation (list numbered from 17 to 
Questions «J ^ .^.^ whether SUMMATIVE evaluation (S) or both 

SUMMATIVE and FORMATIVE evaluation (SF) will assist in attaining 

such aims. 

When evaluating attitudes (domain of affective objectives) all the 
following steps should be taken EXCEPT ONE: 

A. Define explicit behaviours showing a given affective measure. 

B. Define explicit behaviours showing the absence of a given affective 
level . 

C. Observe students in real situations enabling them to manifest the 
behaviours envisaged. 

D Obtain the agreement of a group of experts on the relationship 
between explicit behaviours and the affective level envisagea. 

E. Obtain the student's opinion on the way in which they would behave 
in specific situations. 
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For evaluation the 
quality is 



essential 



VALIDITY 




Do not forget that for an 
educational system 
considered as a whole 

it is its 



RELEVANCE 



that is of primary importance 
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Answers suggested for the exercise on pages 344 to 346 



Question 


Suggested 
answer 


If you do not find the correct answer, 
consult the following pages again 


1 


B 


305, 323, 337 


2 


E 


316-317 


3 





302-303 


4 


E 


318 


5 




322 


6 


A 


> 




7 


B 






8 


C 




\ 316-317 


9 


B 






10 


C 


J 




11 


D 


313 


12 


B 


335-336 ^ 


13 


C 


335-336 


14 


D 


336-339 


15 


B 


336-339 


16 


' A 


318-323 


17 


SF 






18 


SF 






19 


SF 






20 


S 












} 316-317 


21 


s 






22 


SF 






23 


SF 






24 


S 


- 




25* * 


E 


323, 326-327 
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TEST AND MEASUREMENT TECHNIQUES 
(DOCIMOLOGY) 



] 



) 
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This fourth chapter reviews a certain number of measuring instruments, their advantages 
,and limitations, and how they should be used* It shows the wide range of such instruments. sna- 
the importance of choosing them in accordance with the educational objectives to be measured. 
It is stressed that every teacher should make a certain minimum of statistical calculations 

when he carries out an item analysis or calculates the Minimum Pass Level (MPL) of a test* 

\ 

Those with a deeper int-erest in these problems are strongly advised to consult the 
following publications: 

<fr Public Health Papers - WHO 

^ No. 36 A review of the nature and uses of examinations in medical education, 1968 2 
^ No. 52 Development of educational programmes for the health professions, 1973 

* 

After having studied this chapter and the references indicated, you should be able to: 
0 Define the following terms: prerequisite level test, comprehensive pre-final, partial 

or A pre-test, indicate their purpose and their grading. 
^ List the good and the bad features of a test. 
^ Compare the advantages and disadvantages of tests. 
^ Draw, up an ,f essay question' 1 with criteria for objective scoring. 

Construct so-called "objective" questions permitting measurement of a high level 

cognitive process. 

^ Indicate the advantages and drawbacks of the -programmed examination. 

.Compare the advantages and limitations of relative and absolute criterion testing. 
Do an item analysis - calculate the minimum 'pass level for a given test. 
Fill in a specification table for an examination. 



\ 
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DOCIMOLOGY 



Docimology or the study of tests used in measurement techniques is a fairly new 
science. It was introduced into the medical teaching world about twenty years ago again** 
some opposition and the problem has certainly aroused the interest of teachers; however, 
some of them feel that the evaluation "specialists" are trying to poach on their preserves, 
and that this will limit their academic freedom. This is often due to a lack of 
information. 

This lack of information produces certain defence mechanisms in teachers when the 
problem is tackled scientifically, making for more heated reactions. 




Examens et Docinologie, Henri Pieron, Collection SUP, Presses universitaires de France, 
deuxieme edition, 1969. (See index, page 606.) 
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Basic considerations concerning examinations 



Analysis of the most commonly used tests shows that sometimes, often even, the questions " 
set are ambiguous, unclear, disputable, esoteric or trivial. It is essential for anyona 
constructing an examination test, whether a traditional written examination or an objective 
test, to submit it to his colleagues for criticism so as to make sure that its content is 
relevant (related to an educational objective) and of general interest, and does not 
exclusively concern a special interest or taste of the author; that the subject is interesting 
and real for the general practitioner or for physicians with a specialty different from that of 
the author; and that the questions (and the answers in the case of multiple-choice questions) 
are so formulated that experts can agree on the correct response. It is clear that a critical 
review of the question along these lines would avoid the over-simplification of many tests 
which only too often justify a conclusion such as: "the more ycu know about a question the 
lower will be your score". 

f 

The author of a question is not the best judge of its clarity, precision, relevance and 
interest. C ritical review of the question by colleagues is consequently essential for 
a we 11- cons true ted test . 

Moreover, an examination must take the factor of practicability into account, and this 
will be governed by the time necessary for its construction and administration, scoring and 
interpretation of the results, as well as general ea3e of use. | 

If the examination methods employed become a heavy burden on the teacher because of their 
impractical nature he will tend to assign less importance than it warrants to the measuring 
instrument, 

★ N 
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QUALITIES OF A TEST 



* Directly related to educational objective 

* Realistic & practical 

* Important & useful 

* Completebut brief 

* Precise & clear 



★ 



and study again (p, 338) the relations between characteristics 
of an examination. 



Considerations of the type of competence a test purports to measure 



No test format (objective, essay or oral) has a monopoly on the measurement of the higher 
and more complex intellectual processes. Studies of various types of tests support the view 
that the essay and the oral examination, as commonly employed, test predominantly simple 
recall and, like the objective tests in current use, rarely require the student tc engage in 
reasoning and problem- solving. In short the form of a question does not determine the nature 
of the intellectual process required to answer iu . ^ 

Second, there is often a tendency to confuse the difficulty of a question with t .e 
complexity of the intellectual process measured by it. However, it should be noted that 
a question requiring simple recall may be very "difficult" because of the esoteric nature of 
the information demanded; alternatively, a question requiring interpretation of data or 
application of principles could be quite "easy" because the principles of interpretation are 
so familiar and the data to bo analysed so simple* In short, question difficulty and 
complexity of instructions are not necessarily related to the nature of the intellectual 
process being tested . 

Third, there is often a strong inclination to assume that any question which includes 
data about a specific case necessarily involves problem-solving, whereas, in fact, "data" are 
often merely "window dressing" .when the question is really addre: sed to a general condition 
and can be answered equally well without reference to the data. Or, the data furnished about 
a "specific case" may constitute a "cut-and-dried" , classical textbook picture that, for 
example, simply requires the student to recall usually symptoms associated with a specific 
diagnosis* It is interesting to note that questions of this type can be readily converted 
into problems that do require interpretation of data and evalu a tion- simply by making the case 
material conform more closely to the kind of reality that an actual patient, rather than 
a textbook, presents. 

In short, just as each patient in the ward or out-patient department represents a unique 
configuration of findings tnat must be analysed, a test which purports to measure the student's 
cliniuil judgement and his ability to solve clinical problems must simulate reality as closely 
as possible by presenting him with specific constellations of data that are in some respects 
unique and, in that sense, are new to him. 
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STAGES OF 
ASSESSMENT 

t - Pre-req uisite level testing 

> 

- Pre- testing 

Pre -final -feedback -comprehensive testing 

"Subjective" impression 

- Final testing 

- Safety testing 
Follow-up testing 

* 
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different\types of examinations during a course and their stages 



★ 

Prerequisite level test 

I Before commencing | a course it is necessary to ascertain whether the students have 
reached a certain level, namely the prerequisite level (refer back to page 116). A teacher 
must specify which knowledge he considers indispensable to ensure that the students assigned 
to him derive maximum benefit from the instruction he has planned for them. This test shows 
whether all the students are at this level or wt ether they are not, in which case coverage of 
this area must be ensured by modifying the proposed instruction to bring them up to this 
level. If this is not done the quality of instruction must suffer. Depending on the number 
of students who need bringing up to this level, the teacher must decide on the type of 
remedy - either reference to books or additional instruction - for the students concerned, 
possibly with the assistance of students who have reached the level and cap be given the task 
of "instructor". As far as possible a "repeat" for all the students should be avoided, since 
this would amount to Ignoring the diagnosis obtained by means of the prerequisite level teat. 

★ 

Pretesting 

j When a given course commences | it is advisable to make sure of the level of the students 
with respect to the cuurse ; on the one hand, this measurement of the starting level will 
permit the assessment of the real GAIN 'at the end of a course, on the other - and this has been 
Shown in real situations - it may be found that some students are already quite advanced as 
regards the objectives envisaged for the course and allowance should therefore be made for this. 
This is a diagnostic test (see page 316). 

★ 

Interval testing 



These tests must be set | as the course proceeds") to give the student the feedback he needs 
in ordei: to know where he stands after a particular period of instruction. The teacher must 
see that these tests are, as far a*, possible, of the same difficulty as the final examination. 
One way cf doing this is to select ^_ random at least three "packets" from a group of questions. 
These three equivalent packets (ai, a2, aj, see diagram on page 409) will be used not only for 
interval testing but also for pretesting (formative), pre-final feedback comprehensive testing 
(formative) and final testing (certifying). Thus, when the student reaches the final 
examination he will not be haunted by the idea of its difficulty; he will have been brought 
up to the necessary level beforehand. 

★ 

Pre** final -feedback comprehensive testing 



This is a test of the formative type set | before the final examination (compi ehensive) of 



a course or the year (pre- final). Its purpose is to inform the student about his level of 
competence (feedback) and it should not be limited to a single subject but should cover a group 
of subjects. This will be facilitated if the school follows an integrated curriculum. If 
it does not, the teacher must include questions from other fields directly relevant to 
understanding of the subject taught, 

★ 

Subjective impression 

Evaluation of this type is carried out on the basis of the teachers* personal knowledge of 
the students after contact with them during the year, he seeks to divide the students into 
three categories, good, average and bad. It would perhaps be preferable to divide them into 
two groups only: satisfactory and unsatisfactory. This evaluation should be carried out 
[before the final examination 5] at the end of the year. For use see "safety testing". 
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PRE-REQUISITE LEVEL 



Q CONTENT 0 



PRE - TESTING 



CONTENT a t) 



INTERVAL TESTING 1 



CONTENT b [) 



INTERVAL TESTING 2 



i 



CONTENT c Q 



INTERVAL TESTING 3 



PRE-FINAL FEEDBACK 
Comprehensive Testing 




SUBJECTIVE IMPRESSION 



FINAL TESTING 



SAFETY TESTING 



FOLLOW UP TESTING 
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Final testing 

These are of different types; oral, practical, traditional written or multiple-choice _ 
questions on patient manag ement problems, etc. , accord ing to the educational objectives to- _ 
be measured and organized, 1 AFTER the end of a course. 1 

★ 

Safety testing 

This should be carried out if there is an abnormal difference between the "subjective 
impression" and the results of the final examinations. If a student who was considered 
satisfactory or good has a bad mark in the final examination, it is essential to re-evaluate 
the situation and not to give the final examination the role of final and arbitrary sanction 
which it has so often had in the past. 

★ » 

Follow-up testing 

T"»*.s is a form of evaluation which is carried out } sometime after completion of the course! 
to de£?x nine the extent to which the student has retained the acquired level of competence. 
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COMMON DEFECTS OF EXAMINATIONS 



J 



> 



Triviality 



Outright error 



Ambiguity 



Conservatism 



Complexity 



U nintended cues 




DOUIHOLOGY 
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COMM>N DEFECTS, OF EXAMINATIONS 



O - \ 

Review of .examinations currently in use strongly suggests that the most common defects 
of testing are: \ 

\ 

(a) the trivia) ty of the questions asked^ which is all the more serious in that 
examination questions can only represent a small sample of all those that could be asked. 
Consequently it is essential for each question to be important and useful; 

t 

(b) outright error in, phrasing the question (or,, in the case of multiple-choice 
questions, in phrasing the distractors and the correct response); 

(c) ambiguity in the use of language which may lead the stjdent # o spend more time in 
trying to understand the question than in answering it; in addit-jn to the risk of his 
giving an irrelevant answer; 

* 

(d) forcing the student to answer in terms of the bias or even the outmoded ideas of 
the examiner, a bias which is well known and often aggravated by the teaching methods 
themselves (particularly traditional lectures); 

(e) complexity or ambiguity of the subject matter taught, so that the search for the 
correct answer is more difficult than was anticipated; 

(f ) unintended cues in the formulation of the questions that result in transparency as 
to the correct answer; this fault which is often found in multiple-choice questions is 
just as frequent in oral examinations. / 
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OUTSIDE FACTORS TO BE AVOIDED 



Complicated instructions 



Over-elaborate style 



Trap questions 



Ability to pass "tests 
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DOCIMOLOGY 



Outside factors 




In constructing an examination, outside factors must not be allowed to interfere with 
the factor to be measured* 

^Ability to understand, instructions 

Some tests have instructions for students on how to solve the problems that are so 
complicated that what is really evaluated is the students' aptitude to understand the question 
rather than their actual knowledge and ability to use it. This criticism is often made of 
multiple-choice examinations in which the instructions for students appear too complicated. 
This complexity is often more apparent than real and disturbs the teacher rather than the 
student. 

★ 

Ability to use words 

The student may «^sguise his lack of knowledge in such elegant prose and succeed in 
influencing the corrector to such an extent that he judges the words and style rather than the 
student* s knowledge. 

Ability to avoid traps 

This interference does not depend on a measuring instrument, but on possible sadistic 
tendencies on the part of the examiner, who, during an examination, may allow himself to be 
influenced by the candidate's appearance, sex, etc. Some candidates are more or less skilled 
at playing on these tendencies. 

Ability to pass tests 

This is a criticism that is generally made of multiple- choice examinations; it may in 
fact be applied to other formJ of evaluation." In oral and written examinations, students 
develop a sixth sense, often based on statistical analysis of past questions, which enables 
them to somehow predict the questions that will be set. 
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, COMPARISON OF ADVANTAGES AND DISADVANTAGES OF DIFFERENT TYPES OF TEST 



Oral, examinations 



Disadvantages 



Advantages 



1* Lacks standardization 



2. Lacks objectivity and reproducibility 
of results 

3. Suffers from possible abuse of the 
personal contact 

4. Suffers from undue influence or 
irrelevant factors 

5. Lacks an adequate cadre of trained 
examiners to administer the 
examination 

6 # Is excessively costly in terms of 

professional time in relation to the 
limited value of the information it 
yields 



1 # Provides direct personal contact with 
candidates 



2. 



3. 



4. 



5 1 , 



6. 



Provides opportunity to take into 
account mitigating circumstances 

Provides flexibility in moving from 
strong to weak areas 

Requires the candidate to supply his 
own formulation without cues 

Provides opportunity to question the 
candidate about how he arrived at an 
answer 

Provides opportunity for simultaneous 
assessment by two examiners. 
(Unfortunately in practice all those 
advantages are rarely used. ) 



Practical examinations 



Disadvantages 



Advantages 



Lacks standardized conditions either 
in laboratory experiments using 
animals or in bedside examinations 
with patientrs of varying degrees of 
coop era tiven ess" 

Lacks objectivity and suffers from 
intrusion of irrelevant factors 



3. 



4. 



Is of limited feasibility for large 
groups 



Entails difficulties in arranging for 
examiners to observe candidates 
demonstrating the skills to be tested 



3. 



4. 



Provides opportunity to test in 
a realistic setting skills involving 
all the senses while the examiner 
observes and checks performance 



Provides opportunity to confront the 
candidate with problems he has^ not met 
before both in the laboratory and at 
the bedside to test his investigative 
ability as distinguished from his 
ability to carry out "cook-book" 
exercises , 

Provides opportunity to observe and 
test attitudes and responsiveness to 
the total situation 

Provides opportunity to test the 
ability to communicate under pressure, 
to discriminate between important and 
trivial issues, to arrange and display 
the data in a final form 



Standardized practical tests can be constructed; 
medicine" in Public Health Papers No. 61, pp. 18-34. 



see 



"Simulation and evaluation in 
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Essay examinations 



Disadvantages 



Advantages 



1, Limits severely the area of the 
student's total work that can be 
sampled 



2. Lacks objectivity 

3. Presents difficulties in obtaining 
consistent judgements of performance 

4#. Provides negligible feedback 

5. Requires excessive time to score 



1'. Provide- opportunity to candidate to 
indicate nis knowledge of and* his 
ability to organize ideas and express 
them effectively in his language 



Multiple-choice questions 



Disadvantages 



1. Requires extended frime to construct 
in order to avoid arbitrary and 
ambiguous questions 

2. Requires adjustment for positive 
scores that may be achieved by chance 
or by guessing 

3. Provides cues that are unavailable 
in practice 



Advantages 



1. Provides objectivity, reliabity and 
validity * 

2. Increases significantly the range and 
variety of facts that can be sampled 
in a given time 

3. Provides opportunity to obtain 
detailed feedback for both student 
and faculty 
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Comparison of M objective| and "subjective 11 examinations 1 



e than the so-called objective test (multiple- 



1. The essay test is usually less reliabl 
choice question, for example). 

2. The objective fcist is usually more valid than the essay test because of its greater 
sampling of content material. ** 

, 3. Essay tests are economical in time and effort when it comes to making them; expensive 
in both of these aspects when it comes to reading and scoring them. The reverse is 
true for objective tests. 

4. Essay tests are cheap from an institutional point of view in that questions may be and 
often are written on the board and students supply their pwn paper for their answers. 

In the making of objective tests especially for large classes, reams of paper are needed, 

5. Essay tests save time when the group tested is small. A class of 10 is much more suited ^ 
to an essay test than a class of 100. 

6. Guessing may be a problem on objective tests just as bluffing may be a problem on an 
essa*' test. 

Both may be used to measure the various outcomes of learning s~ch as the application of 
knowledge to new situations, the solution of problems, the evaluation of material and of 
ideas, etc. The assessment of such processes is not confined to the peculiar domain of 
the essay test provided that the essay test builder is skilled in the art of test item 
writing. 

8. In the past it was claimed that a student studied one way for an objective test, in 

another fashion for the essay test. This was probably true when the objective test did 
nothing but measure the recognition and recall of factual matter. But with the newer 
and better type of objective items in use today built to measure the diverse objectives 
of education, we have no evidence that study habits used in preparation for these 
objective tests are different from those used in getting ready for an essay test. 
Students' study to pass tests. If the tests stress the assessment of various educational 
objectives, these the students will prepare for regardless of the type of test to be . 
taken. 1 

9 # Is the essay test item more life-like and realistic than the objective test item? This 
we have shown to be not true. If anything, the situations set up for objective test 
items are more similar to those encountered in daily living. 

10. Do students prefer essay tests to objective tests? Inconclusive research nas been 

carried on related to this question. Perhaps, the real issue here is whose essay tests, 
whose objective tests. It is the writer' s opinion that students are more concerned with 
the quality and validity of the various test items that they have to answer than they are 
with the basic forms in which the items appear. 



* Downie, N. M. Fundamentals o£ Measurement: Techniques and Practices . New York: Oxford 
University Press, 1967. 

★ 
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TYPES OF WRITTEN EXAMINATIONS 



ESSAY 

SHORT OPEN ANSWER QUESTIONS 
COMPLETION ITEMS 
TRUE-FALSE ITEMS 
MULTIPLE CHOICE QUESTIONS 

— one best response type 

— the multiple true false type 

— the matching type 

— the comparison type 

— with diagrams, photographs etc. . . 

< 

— in relation to taxonomic level 

— programmed test, patient management problem 

★ 
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The use of the "essay question" j 



The following procedure, is preferable: 

- Use "essay, questions" only to evaluate a type of achievement which cannot be measured 
just as efficiently by other methods (synthesis of a group of complex concepts, summary 
of a document, comparison of two .phenomena, cause analysis, finding relationships, 
criticizing the relevance of a concept, formulating a plan of action). 

- Limit tjie problem posed so that it is clearly apparent to the candidate and define the 
structure of the answer. 

- Employ terms that are as explicit as possible, such as "summarize", or "compare", 
"evaluate", "define", "arrange in order", etc. rather than "discuss" or "state everything 
you know", so that all the candidates immediately know what they have to do. 

- Choose problems which call for careful consideration but whose solution can be briefly 
set out in the time allowed. m 

\ 

- Allow no choice among the questions set. 

- Mark papers anonymously. 

- When two or more teachers correct the same test, they should agree on the' scoring 
procedure before the test and correct the results separately . 

_ For every question, set out yourself the elements which , according to you, should appear 
in the answer (scoring procedure). 

- Use a point system of scoring based upon those elements that are expected to appear in 
the answers. 

- Try out the scoring procedure on a few papers. Preferably, have all the teachers then 
read all the answers to a given question; or, if need be, have one of them read all the 
answers to a particular question in all the papers, have another teacher do the same for 
another question, and so on. 

- Score the answers of all the students to one question before going on to the scoring of 4 
another question. 

- Do not form a judgement of a candidate on the basis of only one question but calculate 
for each candidate a summative score based on the reading of several different essays, 
since such scores are more reliable than the score for any one essay taken separately. 



Use of short, open answer questions j 



This involves "series of questions drafted in such a way that the answer calls for 
a predetermined and precise concept". As their name indicates, the answer expected is short 
and can be "expressed in different forms" (open). "Ideally, only one answer is acceptable no 
matter in what terras it is exposed". The author of the questlqn must "define in advance" 
(and in- cooperation with colleagues) the answer called for by the wording of the question* 
If it appears that conceptually different answers will do for a given question then it should 
be reworded until that drawback disappears. 

Correction is simple since the answer has been predetermined; pagination should allow 
the necessary space for the answer below each question. 

Moreover, all the rules concerning essay tests apply. 

ERJC 213 




NOW 



check whether you have applied the principles explained on pages 412-415 and 418; 
prepare one essay type question and three "shorx open answer questions"* In all cases.it 
should be an "open book test"#* Indicate t}*e answer(s) called for in each case in order to 
obtain an objective scoring (identical score given by independent scorers). 



A M Open book test" means that candidates may use any book, document, personal notes; this 
implies that the question must evaluate a level higUer than recall (see taxonomy, page 151 
et seq »)» 
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DIRECTIONS FOR 
WRITING RECALL AND COMPLETION ITEMS 



Omit only key words and only one or two of these, 
absurdities as: 



Completion items can reach such 



causes 



and 



Only the writer of the question knows the correct answeri 

2. " Repeat blank or blanks at the end of the statement. This is simply a matter of 

efficiency, When so placed, the student can complete the statement on finishing the 
reading. If the blanks appear at the beginning of the statement, he has to read to 
the end, go back to the beginning, ^nd sometimes go through the entire statement again. 

3. Try to have items that have only one correct response . This is often impossible and 
leads to the majnr problem in the use of completion items - the subjectivity of grading. 
Such considerations as the following arise: Are all synonyms to be considered in 
grading? Are all synonyms equally acceptable? 

4. Finally, follov; here many of the other rules discussed for true- false items - that is, 
don't lift items directly from the text: be brief, be clear; use quantitative terms if 
possible, etc. 



EXAMPLES OF COMPLETION ITEMS 



SACCHAROSE IS COMPOSED OF 

MOLECULES OF AND 

MOLECULES OF 

IS THE AUTHOR OF 

"INTRODUCTION TO THE STUDY OF EXPERIMENTAL MEDICINE", 
PUBLISHED IN THE YEAR 



ANSWERS 



/ 



Downie, N. M. Fundamentals of Measurement: Techniques and Practices . New York: Oxford 
University Press, 1967. 



practise drawing up six completion items in your own field: 



1. 



2. 



3. 



4. 



5. 
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DIRECTIONS FOR WRITING TRUE-FALSE ITEMS 1 



© 
© 

© 



© 

© 

© 
© 

© 

© 
0 



Write each item so that it contains only one idea. Confusion results when two or w 
more ideas, some true, some false, appear in the same item. 

Do not lift items directly from the text* Sometimes slight changes such as inserting 
a not are made. Thic in no way improves the item. Items should at least be stated 
in a fashion in which the examinee has not seen them before. 

'Take great care to avoid the use of certain words that make it possible for the 
student to respond correctly to the item even when he knows nothing about the 
material. The word * always" calls for a "false" response, *sometimes *or 'usually * 
a "true" response, *never"a false response, etc. If the test-writer is careful to 
distribute words of this nature equally among true and false items, this problem is 
eliminated. Another factor to be considered here is the length of the item. 
Unless the item-writer is careful, long involved statements will tend to be true. 
A wise student will soon become aware of this and mark long items "true" even when he A 
knows nothing about the subject. ' 

Be certain that the language is exact and expressed in numerical terms when 
necessary • . . usually we don*t know the frame of reference from which an 
individual responds to qualitative words. Such words as large and small , many and 
few, more and less have no place in true-false items. 

Don't emphasize the trivial. If the type of examination has anything to do with 
the type of learning that takes place - and it does - this is one way to encourage 
rather useless learning. 

Avoid parenthetical phrases or clauses that have little to do with the central idea 
of the item. 

Avoid negative statements. After an examinee has read several nots and nevers in 
a series of clauses, he becomes confused. He has heard that two negatives make 
a positive, so he tries to evaluate the statement from this point of view and then 
to decide whether it is true or false. 

Have, in the long run, an equal number of true and false statements. These should 4 
be arranged so that there is no particular pattern of response. 

Attribute to someone statements that reflect attitudes, basic philosophies, schools 
of thought, etc., otherwise there is no scorable answer. 

An important inconvenience of this type of question is that the correct answer can 
be found by chance 50% of the time. 




e, N. M. Fundamentals of Measurement. Techniques and Practices. New York: Oxford 



University Press, 1967. 
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EXAMPLES OF TRUE OR FALSE ITEMS 



ENZYME MOLECULES CONTAIN NATRIUM 



THE PROXIMAL PART OF THE POSTERIOR 
BRANCH OF THE RADIAL NERVE TURNS 
AROUND THE NECK OF THE RADIUS 



- NYSTAGMUS IS OBSERVED IN TABLES 




TRUE 





TRUE > FALSE 




TRUE, 




NOW 



practise drawing up six questions of the 
"TRUE-FALSE TYPE" 



in your own field. 



1. 



2.' 



3. 



4. 



6. 









D 


a 


D 




D 


a 


r> 


a 

G 


D 
D 
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DIRECTIONS FOR WRITING MJLTIPLE-CHOICE QUESTIONS 



1. Make certain that the stem consists o£ a statement or an idea, not just a single word. 

2. Place all common elements in the stem of the item. This adds simplicity and 
compactness to the item. 

3. Make each item completely independent of answers to other items. 

4. Eliminate all unrelated details from an item. 

5. In general, avoid negative statements, but if the word "NOT " does appear in the stem of . 
the question, underline it to draw the student's attention to it. 

6* Use plausible logical distractors . Each distractor* should, by its content or nature, 
be such that it appears to have something to do with the question. Unrelated 
distractors appear silly to a thoughtful examinee. As the number of possible answers 
is cut down, the it en loses some of its value, 

i 

7. Avoid the use of clues that may suggest the correct answer. 

8 # Be sure that the distractors and the correct . ^sponse possess homogeneity, that is, they 
should be fairly similar in content or in the total number of words. 

9. Be careful of the use of 'none of the above" as a distractor or as a correct answer. 

10. If it is impossible to obtain more than three plausible responses, don't waste time 
trying to invent some others. 

11. When dealing with items that have numerical answers, arrange the answers in order from 
large to small or vice-versa. 

12. Arrange the place for the correct answer in such a way that, for the test as a whole, no 
letter corresponding to a given answer appears more frequently ttjan some other letter. 
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TEST ITEM CHECK LIST 



It should be possible to answer "yes" to each cf the following questions about a test 



item: 



1. In general 

1*1 Is the item as a whole realistic and practical? 

1*2 Does it deal with an important and useful aspect of the profession? 

1.3 Is it phrased in the working language of the profession? 

1.4 Does it call for professional knowledge? 

1.5 Is it independent of every other item in the test? 
'1.6 Is it specific? 

1.7 Does i't avoid the fault of giving away the best response by irrelevant details or 
extraneous data? 

2. The essential problem 

2.1 Is it clear? 

2.2 Is it stated accurately? 

2.3 Js it stated briefly but completely? 

2.4 Does it contain only material relevant to its solution? 



3. The distractors 

3.1 Are the distractors important, plausible answers rather than obvious distractors? 

3.2 Do they deal with similar ideas or data expressed in similar form? 



Adkins, D. C. , Construction and Analysis of Achievement Tests, Washington D.C. , 
US Government Printing Office, 1947. 
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EXAMPLES OF MULTIPLE CHOICE QUESTIONS 



One "best response' 1 type 




In differentiating cirrhosis of the liver from chronic constrictive pericarditis, a 
useful physical sign is: 

A) hepatomegaly; 

B) ^scites; 

C) distention of the neck veins; 

D) pitting oedema of the ankles and legs; s 

E) 'splenomegaly. 



/ 




Active 



immunization is available against all of the following diseases EXCEPT: 



A) tuberculosis; 

B) smallpox; 

C) poliomyelitis; 

D) malaria 

E) yellow fever. 



< 



The multiple- true-false type 



(also called "multiple-response item") 



This type consists of a stem followed by several true or false statements. The candidate 
is to determine whether or not each of the four statements which follows is true or false. He 
then responds according to a code which permits one out of five possible combinations or 
responses whereby on/ two, three, or all four statements may be true. 

- when properly written, the multiple-true-false item type tests the student's knowledge 
or understanding of several] related aspe.Ls of a substance, a disease, or a process; 

each of the statements or /completions offered as possibilities must be clearly true or 
false. This is\in contrast to the type M a ,f format in which alternatives which are 
"partially correct^vma^r be used as distractors; 

this type of item should be written so that no two of the alternatives are mut ally 
exclusive, i.e. the answer "all are correct" must be a possible response . 
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The directions for this item type are as follows: 

For each of the incomplete statements below, ONE or MORE of the completions is correct. 
; 0n the answer sheet blacken space under: 

A) if only 1, 2 and 3 are correct; 

B) if only 1 and 3 are correct; 

C) if only 2 and 4 are correct; 

D) if only 4 is correct; 

E) if all are correct. 




A child suffering from an acute exacerbation of rheumatic fever usually has: 

1) an elevated sedimentation rate; 

2? a prolonged P - R interval; 

3) an elevated antistreptolysin 0 titre; 

4) subcutaneous nodules. 



The matching type 

Directions for constructing matching items * 

- Limit the number of entries to about 10. If situations arise where 20 or 30 entries 
must be considered, construct two or three matching items. When long lists have to 
be matched, the student wastes too much time in trying to find the correct response* 

- Do not break items by the bottom of the page. The complete item should be on 
the same page. 

- Have a longer list of questions than of possible answers and state in the directions 
that these may be used more than once. When there are an equal number of questions 
and answers, it is possible for the student, after responding to some of them, to 
complete his task by elimination and guessing. 

- Strive for homogeneity. 



Downie, N. M. Fundamentals of Measurement; Techniques and Practices , New York, 
Oxford University Press, 1967. 
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The directions given to examinees for this type of item are as follows: 



"Each group of questions below consists of lettered headings followed by a list of 
numbered words or statements. For each numbered word or statement, select the one heading 
that is most closely associated with it and blacken the corresponding space on the answer 
sheet. Each lettered heading may be selected once, more than once, or not at all. 

Examples : 

A) 
B) 





O 
D) 
E) 



increased metabolic activity;\ 
hyper insulinism; 
lack of storage of glycogen in the liver; 
storage of an abnormal glycogen in the liver; 
decreased secretion by pituitary or adrenal glands; 

4) adenoma of islets of Langerhans; 

5) violent exercise; 

6) hyperthyroidism; 

7) Simmond's disease; 

8) Von Gierke's disease; 

9) epidemic hepatitis; 



A) 
B) 
O 
D) 



Questions 10 to lh 



sodium bicarbonate; 
sodium carboxymethylcellulose; 
aluminium hydroxide gel; 
none of the above; 

10) a gastric antacid which is also used in the therapy of hypoparathyproidism 
because of its property of reducing the absorption of phosphorus; 

11) because it is absorbed, it may cause alkalosis, particularly in infants and 
elderly patients; 

12) a gastric antacid which has the disadvantage of causing "acid-rebound"; 

13) a gastric antacid which precipitates and inactivates gastric pepsin; 

14) a gastric antacid and demulcent which can be converted to liver glycogen. 



The comparison type 



The "comparison" type permits one. to compare and contrast two diseases, signs, symptoms, 
laboratory findings, etc. / 

When using this type of item, one must be careful to; (1) avoid the trivial; (2) avoid 
selecting as one of the pair something that is rare or unusual. For example, if the item 
asks about the relation of a certain symptom to disease "x" or ,f y M , and the frequency of the 
symptom in the tuo diseases is 90% and less than 1% respectively, then the examinee is in a 
dilemma. If he follows the principle of the "general rule", he may select answer A ("x" 
only); but if he is aware that the symptom does occur in the exceptional case of disease "y", 
then he may select answer C ("both"). Which response is correct? 
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The instructions for this type of item are as follows: 

"Each set of lettered headings below is followed by a list of numbered words or phiases. 
For each numbered word or phrase, blacken the space on the answer sheet under: 

A) if the item is associated with (A) only ; 

B) if the item is associated with (B) only ; 

C) if the item is associated with both (A) and (B); 

D) if the item is associated with neither «{A) nor (B) . 



A) hookworm disease; 

B) ascariasis; ' 

C) both; 

1)) neither; 

15) eosinophilia; 

16) hypochromic anaemia; 

17) infection through the skin. 

. Questions with diagrams, photographs, etc . 



Questions 18 to 20 





Level of 




Question 18 



Total n«ng capacity 



Question 19 



Inspiratory capacity 



Question 20 



Vital capacity 
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y Practise drawing up four multiple choice questions in the cognitive field using the 
principles just defined (pages 428 to 433). At least three of them should measure an 
intellectual process above level one M simple recall M (they should be on level two "interpret 
tion of data" or level three "problem solving 11 ). For this purpose use the objectives you 
have drawn up on page 165. 



Now refer to pages 428 and 429 anu 0 ee whether your questions comply with the principles 
advocated. You can also consult the following examples (pages 437 and 440). 



QUESTION 1. 



QUESTION 2. 



QUESTION 3. 



QUESTION 4. 
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EXAMPLE OF ITEMf CONSTRUCTED IN 
ACCORDANCE WITH THE TAXONOMIC LEVEL 



(cognitive domain - see pages 152-155) 



( 



Utilization of clinical tests for application of basic science concepts 



D 



LEVEL 1 - Recall of facts 



r 



/ 



Item 1 : | The Fick principle is illustrated mathematically by which of the following 
formulae: 

_VX- 

0 Consumption (ml/min) 

B . Cardiac Output (litres/min) 
KearK Rate (beats/min) 

C . Heart Rate (beats/min) 

Cardiac Output (litres/min) 

$ . 0 Consumption (ml/min) 

aT - V„ _ 
°2 °2 

E . Cardiac Output (litres/min) 
Body Surface Area (sq. metres) 



LEVEL 2 - Interpretation of specific data 



Item 2 : J Apply the Fick principle to the following data and calculate the amount of 
glucose (in mg/min) extracted by the spleen of a dog: 

Cardiac output - 2 litres/min 

Stroke volume - 200 ml/min 

Cardiac index - 2.8 litres/sq. metre 

Arterial glucose - 100 mg/ml 

Splenic vein glucose - 96 mg/ml 

Splenic blood flow - 50 ml/min 
Which of the following corresponds to the result of your calculations?: 
A) 140 B) 178 C) 200 D) 225 E) 250 
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LEVEL 2 - Interpretation of specific data 



(continued) 



Clinical case (for questions 3, 4 and 5) 

A 25 year old male patient comes to your office complaining of cramps in the muscles of 
his arms or legs whenever he engages in strenuous musculo exercise. This complaint has 
become more prominent over the past three co four years. The general physical examination is 
normal. The patient has previously been seen by another doctor who performed a muscle 
biopsy. When you call the other doctor's office, you learn that the only abnormality noted 
on the biopsy was excess accumulation of glycogen in the muscle fibres. Electromyography 
was normal. 

9 

The blood chemistry determinations related to carbohydrate metabolism which are performed 
in the hospital where you work are glucose and lactate. 

You decide that you want to find out why the patient has muscle cramps and excess glycogen. 
To obtain further data, you apply the Fick principle to the right leg and insert small polyethy- 
lene catheters in the femoral artery and femoral vein at the level of the inguinal ligament. i 
You make a blood flow estimation for the right leg and obtain a value of 200 ml/min at rest. 
You also take two resting blood samples from the arterial and venous catheters. 

You then have your patient pump a bicycle ergometer for five minutes, and at the end of 
exercise you take a series of five arterial and venous samples at two minute intervals. You 
have glucose and lactate determinations performed on each of the 14 blood samples. The 
results on the resting samples are reported as follows: 





Arterial 


Venous 


Glucose 


75 mg % 


73 rag 7. 


Lactate 


8 rag % 


9 mg 7. 



Item 3 :J The glucose and lactate extraction mg/min) by the tissues of the right 
. leg in the resting state is: 





A 


3 


r* 


D 




Glucose 


400 


4 


100 


0.01 


40 


Lactate 


200 


2 


200 


0.005 


20 
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- Solution of a problem 



On the basis of your knowledge of muscle carbohydrate metabolism, rhe 
history from the patient and the muscle biopsy report, which of the following 
sets of pre-post exercise data on venous blood would be most likely in this 
patient? " 



200 mg % 





Rest 


Ex. 


Rest 






Min. 


Glucose 




mg % 






Rest | 


Ex. 


Rest 


i 




Min. 



20 mg % 









Rest 


Ex. 


Rest 






Min. 


Lactate 




mg % 






Rest 


Ex. 


Rest 






Min. 



Glucose 



Lactate 



^200 mg % 



Glucose 



20 mg % 







Rest 


Ex. 


Rest 


Rest 


Ex. 


Rest 




| Min. 




Min. 



Lactate 



J 



200 mg % 









Rest 


| Ex. 


Rest 


1 


Min. 



Glucose 



20 mg % 





Rest 


Ex. 


Rest 






Min. 



Lactate 



Cj 



AW 



/ 



*k LEVEL 3 - Solution of a problem 



(continued) 



Item 5 



Study the basic principles of carbohydrate metabolism as shown in the figure 
below. Note those parts of the metabolic route indicated by a letter (A, B, 
C, D and E) . 

Assuming that one of several reasonable explanations of the symptoms presented 
by the patient is a metabolic block, and on the basis of the data available to 
you after study of the preceding parts of the question, select the letter 
which would be the most probable site in this route of metabolic block in the 
patient. 

Mark on the answer sheet the letter identifying the point in the route where a 
more thorough study would have most chance of giving positive results. 



Glycogene 
Uridine 



Carbohydrates 



Glucose 

in Glucose 



CSF 




Di-Phospho 
Glucose 



A 



ATP-ADP 



Hexokinase 



wGlucose-l-PO 



/ 



Phosphorylase 




Cell Wall 



Glucose-6- 
Phosphates 



0 



Glucose-6-PO, 



It 



Hexosemo no phosphate 
Shunt 




Fructose-6-P0 



ATP-ADP 



Phosphofructokinase 



0 



Fructose-1, 6-Di phosphates 



Fructose-1, 6-Diphosphate 



It 



(2 mo Is per raol of glucose) 
Pho s phog ly c er a ldehyde 



- GLYCEROL 
(from fats) 



ADP 
DPN 



-ATP 
-DPNH 



Phosphoglyceric acid 

♦ t 

Enolphosphopyruvic acid 



ADP 



-ATP 



DjNir: 



:dpn 



Proteins: 




: Alanine : 



rPyruvic acid: 



rLactic acid 



Aspartic 
acid-*- 



Acetyl-CO A 

•Oxaloacetic 



acid 



Fats 



Citric acid 
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Citric acid 
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THE PROGRAMMED EXAMINATION 



The advantages of this relatively recent method (it appeared at the beginning of the 
1960s) are so great that they should help in o/ercoraing the difficulties attaching to its use. 

Briefly, its aim is to measure (by simulation on paper) the "clinical competence" of the 
physician. * 

Like the multiple choice questions it is highly objective and can be corrected by 
computer. Its name shows that whis new examination method has certain aspects identical to 
those of programmed teaching, where the candidate advances, step by step, through a series of 
consecutive clinical problems. 

The method was developed in the United States of America and several types of such 
simulation test can be found in the literature. They are sometimes referred to as Patient 
Management Problems (PMP), Clinical Simulations, etc. 

Objectives of the method 

The aim of the method is to evaluate a physician's clinical competence: 

- Can he detect and satisfactorily interpret abnormal signs and symptoms? 

- Does he then reach a reasonable diagnosis? Is his judgement satisfactory in choice 
of treatment? 

Until the early 1960s examiners tried to find an answer to these questions by confronting 
the candidate with a carefully selected patient. This method was effecti- e in the past, when 
candidates were riot very numerous. More recently, faced with, thousands of candidates, 
thousands of patients and thousands of examiners test specialists confronted a difficulty 
which they rapidly recognized. There were three variables; the candidate, the patient, and 
the examiner* This represented two variables too many for a valid eval u ation of the 
candidate . 

Research consisted in seeking a valid definition cf the qualities involve^ in what is 
termed clinical competence (at the level, for example- of a hospital intern). One method 
employed was that of the questionnaire using Flanagan's "critical incident" technique (see 
p. Ill ). 

Through direct interviews and questionnaires, several hundred physicians were asked to 
describe clinical situations during which they had personally observed interns in the course 
of their work and had been impressed on the one hand, by examples of satisfactory clinical 
conduct, and, on the other, by examples of unorthodox clinical conduct. Several thousand 
situations of this type were analysed. This ample documentation gave an idea of what had to 
be evaluated. 

The following step was tu determine how to evaluate this "what". Numerous methods were 
envisaged. 

Silent films, in colour, of carefully selected patients were used instead of actual 
patients, the examiner being replaced ty a series of multiple choice questions concerning the 
patient presented. This method proved satisfactory and it is now in routine use by examining 
bodies. 



* See also "Simulation in instruction and evaluation in Medicine" in WHO, Public Health 
Papers, No. 61. 
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Finally another method was found (programmed .sting) for evaluating the abilities of the 
intern when placed in a clinical situation as rea\ as possible and called upon to face the 
unforeseeable problems presented by every patient. 

rn everyday routine the intern may be required, for example, to see a patient who has just 
been admitted to the medical department. He goes to the patient, gets information from him 
and makes a clinical examination. He must then take a certain number of decisions. He calls 
for certain laboratory tests whose results, combined with those of the clinical examination, 
will lead him to reach a diagnosis and decide on a treatment. The patient's condition may 
then improve, worsen or remain unchanged by the treatment. The situation changes, new 
problems appear and fresh decisions must then be taken in the light of these new data. 

PROGRAMMED TESTING recreates, as far as possible, the changing situation represented by 
every patient. Each patient is described in accordance with a real case history. From four 
to six "clinical problems" are presented following the case study with the aim of simulating a 
situation changing in time. The patient can be followed up for several days, weeks or 
possibly months, just as in real life, until he is discharged either cured or with his 
condition improved or, if he dies, passes to the autopsy table. 

| At each step in time. the candidate is required to make decisions; "he immediately learns 

the results thereof and, with this fresh information, goes on to the following "choice", 
always concerning the same patient. 

The "eraser" technique 

The raetho'dology of this type of test, as with programmed teaching, implies that the 
information given to the candidate is hidden from him until he has made a decision and thus 
becomes entitled 10 obtain additional information. 

We shall not deal at length with the different technicaj." difficulties that had to be 
overcome before a satisfactory system was found. As things are at present the appropriate 
information is hidden by a completely opaque layer of ink which can be removed, however, with 
an ordinary pencil eraser, or revealed by a system comparable to invisible ink. 

The method can be easily used for examining a large number of candidates simultaneously. 
Examples^of case histories "| 



A clinical observation is described to the candidate and he is ther asked: 
1 



To study the details carefully and then the list of possible decisions presented for 
each "choice" linked with the initial observation. 



2. To choose from the list only the numbered items which seem important and appropriate. 

3. To erase the corresponding opaque rectangle on the. answer sheet. 
The candidate is reminded: 

1. That it is not suggested how many proposed decisions he should choose. 

2. That information will appear in the space erased for both correct and incorrect 
choices . 

3 That since the information gradually revealed may orientate his subsequent decisions 
he should consider them one after another in the order indicated. 

A But that within each "choice", the order of the numbered decisions is proposed at 

random although it is advisable tor the candidate to re-establish a logical order m 
his choice. 



231 



* . " JWUIHULOM 



-Scoring 



The usual machine method of scoring is emplojed, each space erased corresponding to one 
response so that the candidate is unable to cancel a mistake once his choice has been made 
(the same applies in the case of a real patient). 

The candidate is penalized whenever he makes an incorrect choice <?nd whenever he fails to ' 
make a choice which was appropriate. The scoring is thus negative, taking into account sins 
of both omission and commission. 

The choices proposed to candidates can be divided into three groups: ""^"^ 

(a) Appropriate; should be made with the aim of improving the patient's condition; 

(b) Not indicated; should not be done and, if it is done, may be dangerous for the 
patient; 4 

(c) Neutral, of debatable importance; may or may not be done according to local 
conditions, teaching, customs, " etc. 

The candidate who does not make a choice regarded as suitable by the examiner or who makes 
a choice regarded as not indicated or dangerous is penalized. 

Choice (c) has no effect on the scoring. 

Consequently, this is a scoring system* completely different from the multiple choice 
question method, where the candidate must select the best (and only ) answer from several 
suggested. 

In programmed testing he must decide to select all those choices he regards as appropriate 
for the treatment of the patient. He is not told how many choices he must make. The same 
applies in medical practice, where the physician makes a choice between what should be done 
and what should not be done. If he is proceeding on the right lines, he makes a certain 
number of decisions out of all those which could be made. 

Experience (immediate feedback after "erasi-ig") gives him fresh data which will guide 
him towards new decisions. 

If he is on the wrong track, experience ("erasing") will show him his errors as they 
arise and give him a cha^-e of changing his action although he will be unable to cancel put 
hi3 mistakes. 



IMPROVEMENT OF CASE HISTORIES 



By means of dociraological correlation studies the quality of questionnaires can be 
improved. The teachers who have drawn up the questions learn from the statistical study, 
question by question, how they can better test discriminatory qualities of judgement enabling 
a choice to be defined as "appropriate", "non-indicated", or "neutral". 

The task is different and considerably more arduous than that involved in drawing up the 
usual multiple choice questions. 

On the other hand, the examiners find themselves on more familiar ground and feel that 
they are dealing with practical clinical situations in a much more realistic way than when they 
had to decide on a single best choice. 



The method is far from perfect and calls for constant improvement, but gives new hopes for 
the evaluation of the clinical competence of physicians. It makes possible the evaluation of 
certain qualities which were not evaluated i.: the past, qualities considered essential for 
preparing the physician to assume independent responsibility in the practice of his protession. 
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Other scoring systems have been described. 
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A If -you would like to try this exercise, cover up the right hand column giving the 
*7 answers. 

OBSERVATION 

I. A man 57 years of age and in good health until then is referred to the out- 
patient department because three days ago he suffered from anterior chest pains of two 
hours duration and, an hour previously a repeated, less violent pain lasting a quarter 
of an hour. On examination the, patient is found to be in good general condition. 
He weighs 95 kg and measures 1.60 m in height. He is no longer in pain. The pulse 
rate is 90/min, and blood pressure llo/?0 (usually 18o/lOO). Lungs normal, 
peripheral pulses felt, liver extends one f ingerbreadth below the costal margin, not 
tender, nothing abnormal discovered in lower limbs. Urinalysis on admission: 
sugar -H-, acetone 0, albumin -f. The patient is then hospitalized. 



ACTION TO BE TAKEN 


ANSWERS 


SCORE 1 


1. Take an emergency electro- 
cardiogram 


1. Sinus rhythm PR 0.22 

Left bundle branch block 
and deviation upwards of ST 
in VI, V2, V3. 


+1 


2. Chest X-ray with patient in bed 


2. Normal heart volume. j 

Aorta somewhat enlarged, lungs 
normal. 


0 


3. Transaminases 


3. SGPT 15, SCOT 40 


+1 


4. Emergency determination of urea, 
blood sugar, blood bicarbonate 


4. Urea 0.60 g, blood sugar 2 g/l, 
blood bicarbonate 23 m£q/l 


+1 


5. Anticoagulant therapy (intra- 
venous drip heparin), oxygen 
therapy, monitoring of pulse 
and blood pressure 


5. An hour later the patient is 

well, the pulse rate is 80 and 
the blood pressure 130/80. 


+1 



The scoring shown for this observation was decided on by clinicians who were kind 
enough to give their assistance and whom we should like to thank. Other clinicians might 
perhaps prefer different scoring or a different therapeutic approach. The purpose of the 
article is not to discuss this, but to make clear the general principles of the method. 




ACTION TO BE TAKEN 


ANSWERS 


SCORE 


6. Intravenous digitalin, 
sulfonamide diuretics 


6. Treatment cancelled by head 
of department 


-1 


7. Intravenous levophed 


7. Sudden rise in blood pressure 
to 25o/l40. Acute pulmonary 
oedema 


-1 




II. The next day, while he seems to be doing well, the patient loses consciousness 
during medical rounds without any warning pain or spasm. He is pale, blood pressure 
too low to measure, pulse imperceptible and heart sounds inaudible* 




ACTION TO BE TAKEN 


ANSWERS 


SCORE 


8. Intracardiac adrenalin 


8 . Ine f f ec tive, death 


-1 


9. Intravenous ouabain 


9. ' Death 


-1 


10. 20 units intravenous Insulin + 
500 ml saline with bicarbonate 
by intravenous drip. 


10. Death 


-1 


11* Emergency electrocardiogram 

before undertaking any treatment 


11. P wave 70/mn. Rare ventri- 
cular complexes, the patient 
dies during the recording 


-1 


12. External heart massage with 
mouth-to-mouth respiration 


12. After 15 minutes the patient 
regains colour, pupils return 
to normal, the femoral pulses 
are easily perceptible, blood 
pressure 180, return to 
consciousness 


+1 


13. Intravenous procain amide 


13. Definitive heart arrest, death 


-1 


14. Solucamphor 


14. Ineffective, death 


-1 
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III. An hour after the heart attack the patient is again normally conscious and 
seems to be in good condition. The heart rhythm is 40, blood pressure 150/60. The 
electrocardiogram shows a ventricular rhythm of 35/min with 70/min P waves independent 
of the ventricular complexes, the latter having the appearance of a left bundle branch 
block. 

Respiration is normal, diuresis is 30 ml since loss of consciousness. The 
humoral balance is as follows: blood sugar, 2.20 g/l; bicarbonates, 15 mEq/l; 
ketone bodies, 0; sodium, 140 mEq/l; chloride, 100 mEq/l; potassium, 5.4 mEq/l. 



ACTION TO BE TAKEN 


ANSWERS 


SCORE 


15. 250 ml of molar sodium lactate 
solution administered by 
intravenous drip at the rate of 
80 drops per minute 


15. Increase of heart rhythm to 
80, followed by further 
syncope, ventricular fibril- 
lation, and death , 


-1 


16. Isopropylnoradrenalin, 1 mg 
every two hours 


16. Increase in heart rhythm to 
55 per minute. No syncope 
during eight days 


+1 


17. No treatment, patient kept under 
surveillance 


17. Repeated syncope leading to 
syncopal status. Collapse, 
arrest of respiration. Death 


-1 


18. Emergency introduction of an 
incorporated pacemaker 


1 ft fprp^ 1 pmKrjl l when 

recovering from the operation 
three days later 


- 1 


19. 500 ml of 147, bicarbonated 
saline 


19. 12 hours later, alkaline 
reserve 23 mEq/l 


-1 


20. Insulin, 48 intravenous units 

to be repeated depending on 
« glycosuria 


20. Hypoglycaemic coma 


-1 


21. 500 ml of 10% mannitol 

administered over 12 hours 

1 


2i. Diuresis during the 24 hours 
following the commencement of 
intravenous drip 1500 ml 


0 
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IV. Af£er a week the heart rhythm is stable and regular at 70/min. The electro- 
cardiogram shows a complete le*t bundle branch block, the patient is in good 
condition, weight 83 kg; blood pressure 140/90; glycosuria +; fasting blood sugar 
1.30 g; blood sugar an hour-and-a-half after midday meal, 1.90 g, urea clearance 
45 ml/min; albuminuria, 1 g per 24 hours; acetonuria, 0. 



ACTION TO BE TAKEN 


ANSWERS ^ 


— SCORE 


22. Patient authorized to get up and 
sent home for a fortnight's 
convalescence 


22. Three months later, dies ^* 
denly in the street; post- 
mortem: ruptured dilatation 
^ of^left ven ricular wall. 


-1 


23. Kept in bed for another three 
weeks, continuing anticoagulant 
treatment with vitamin K 
antagonists 


23. Favourable course 


+1 


24. 1200 calorie diet 


24. Well tolerated, disappearance 
of glycosuria 


+1 


25. Restricted carbohydrate diet 

with hypoglycaemic sulfonamides 


25. The patient has gained 2 kg 

, after three weeks ^ 


i 

f 

0 


26. Diet with 250 g of glucides 

as well as 20 unitj per day of 
protamine zinc insulin 


26. Numerous hypoglycaemic episodes; 
during one of them, stroke with 
some cerebral softening 


-1 
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V. A month after the onset of the illness, during convalescence, fever of 38-38. 5°C, 
appears, chest pains radiating into the shoulder and left arm unassociated with 
effort. Regular heart rate of 75, systolic and diastolic rubbing sound on ausculta- 
tion. No pain in the lower limbs. Slight dullness on percussion of left base. 



ACTION TO BE TAKEN 


ANSWERS 


SCORE 


27. Repeat electrocardiogram 


27. Left bundle branch block, 
PR 0.16 seconds, deviation 
upwards of ST in the 
^ precordial leads 


+1 


28. Chest X-ray 


28. Left pleural effusion 


+1 • 


29. Transaminases and sedimentation 
rate 


29. SCOT 10, SGPT 16, Sed. Rate 
60, 100, 120 


+1 


30. Patient again ordered complete 

bed rest for a month without any 
other treatment 


30. Increased sedimentation rate 
and pain persists for three 
months 


-1 


j 31. Reinforce anticoagulant treat- 
ment by intravenous heparin 


31. Pericardial effusion with 
tamponade 


-1 


32. Corticoid therapy, 30 mg per day 
for eight days with aspirin as 
replacement and monitoring of 
glycosuria 


32. Disappearance of fever and 
pains in a few days as well 
as of pleural effusion and 
rubbing sound 


+1 


33. Pleural puncture 


33. Sero-f ibrinous exudate 

obtained. On following day, 
extensive haemothorax and 
shock 


-1 
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VI. The patient has recovered and is convalescing. He now has to be given long- 
term advice concerning general behaviour, diet and medication. 



ACTION TO BE TAKEN 


ANSWERS 


SCORE 


34. Complete cessation of occupa- 
tional activities for/a very 
long period / 


34. Development of a depressive 
syndrome with general feeling 
of ill-health and anxiety 


-1 


35. Strict salt-free diet with 
long-term administration of 
diuretics. Administration 
of digitalin at intervals: 
10 drops, four days a week 


35. Vomiting and dehydration 
symptoms owing to lack of 
salt, hypokalaemia with 
varying ventricular extra- 
systoles 


-1 


36. Maintenance of anticoagulant 

treatment for a lengthy period 
with fortnightly monitoring of 
prothrombin level. Normal 
occupational activities at the 
end of the second month. 
Prohibition of tobacco. 
Continuance of diet previously 
prescribed 


36. Regular outpatient visits; 

patient getting on well, works 
full-time, no glycosuria, has 
lost several kg in a month 

- 


+1 



* 
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DESIRABLE QUALITIES OF RATING SCALES 



* Clarity * Variety 

* Relevance * Objectivity 

* Precision * Uniqueness 

* 

ERRORS TO BE AVOIDED IN RATING SCALE CONSTRUCTION AND USE 



Errors of leniency 
Error of central tendency 
Halo effect 
Logical error 
Contrast error 
Proximity error 
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Factors influencing rating 



1 



It is supposed that a human observer is able to carry out a more or less objective 
evaluation, and it is known that such evaluations are subject to error. 

— Leniency is a well-known factor. One means of counterbalancing this tendency is 
to propose a scale which comprises only one "unfavourable" appraisal in five, for instance: 



1 



Poor — Average — Good — Very Good - Excellent 
In this case, the appraisals will probably be distributed symmetrically around 



"good"- 



Examiners have a tendency not to give extreme appraisals and hence to group all 
candidates around the mean. This central tendency may be reduced by using a scale which is 
wider at the centre than at the ends, for example: 



-7 -6 -4 0 -*4 -f6 4-7 

» A particular feature of a candidate sometimes seems so important to the examiner 
Chat it influences the overall evaluation. Thorndike called this the Halo Effect . However, 
this effect is reduced as the number of separate aspects of the problem dealt with by the 
evaluation is increased. 

r The " Logical" error is similar to the halo effect and it occurs when the examiner 
supposes that there is a relationship between two variables to be evaluated and that "if the 
first variable is of a particular order, the second will be similar". This error may be 
reduced if the evaluation relates to an observable element rather than to an abstraction, 
which could lead to semantic confusion. 

m An observer who is very orderly will have a tendency to consider, by contrast , 
that other people are less orderly than he is, and vice versa . On the other hand, people 
frequently believe that "others are like me" and are very surprised to see that this is not so. 

m If an observer evaluates two different factors, the evaluation of the one factor has 
a tendency to influence that of the other, and the shorter the interval between the two 
evaluations, the more pronounced the tendency will be ( proximity error ) . 

» This list of human factors influencing evaluation is not restrictive but those 
mentioned are the ones best known. 



★ 



1 Guilford, Psychometric Methods , pages 278-2GO 
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TEST CONSTRUCTION SPECIFICATION TABLE* 



CONTENT 


PROCESS 








Re c a 1 1 of 
facts 


Interpretation 
of data 


Problem 
Solving 


% 


n 


OBJECTIVE 1 


6 


3 


1 


20 


10 


OBJECTIVE 2 


8 


2 


0 


20 


10 




12 


6 


2 


40 | 20 


OBJECTIVE "n" 


4 


4 

* 


2 


20 


10 




No. of items 


30 


15 


5 




50 


% 


60 


30 


10 


100 



As an exercise, fill in a specification table for the last examination 
you were responsible for 



see p. 253-254 
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TEST CONSTRUCTION SPECIFICATION TABLE 



t 



v 



CONTENT 

* 


PROCESS 


• 






! 
i 

i 
1 


Recall of 
facts 


Interp retatio n 
of data 


Problem 
Solving 


% 


n 


1 

OBJECTIVE 1 












OBJECTIVE 2 












! 

i 




i 

I i 
i i 

i i 

'i 






OBJECTIVE "n" 














/No. of items 












% 








100 
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RELATIVE and ABSOLUTE criteria tests 



These two* expressions are also referred to in the literature under norm-referenced and 
criterion-referenced tests. 

It is very important to distinguish between tests based on reference to the "norm" (i.e. 
in accordance with the curve for the results of all the students who have taken the same test, 
and that is why this criterion is termed RELATIVE), and tests based on reference to a 
"criterion" (i.e., in relation to the description of an acceptable performance namely the 
specific educational objective fixed in advance). 

Norm-referenced tests are the ones most frequently employed. Unfortunately their 
disadvantages greatly outnumber their advantages, for if a group of students is particularly 
brilliant the utilization of relative criteria tests will lead to some of them being failed 
♦although their level of performance may be satisfactory from the absolute viewpoint. On the 
other hand, if a given group of students has on the whole a low performance level (because 
the appropriate instruction has not been given, or has been poorly given, or for some other 
reason) the relative criteria system may enable "poor" students to pass if they are above the 
average of their group. u 

If it is felt unanimously by an evaluation committee, for example, that all the students 
should be able to master an emergency procedure, then this can be ascertained by an absolute 
criteria test. 



A measuring technique adapted to absolute criteria tests is suggested 
on pages 458-460, namely calculation of the Minimum Pass Level (MPL) . 

You will find on pages 465-473 the measuring techniques suitable for 
relative criteria tests (difficulty and discrimination indices). 



\ 



Calculation of the Minimum Pass Level 
for a MCQ test 



1. DEFINITION 

The Minimum Pass Level (MPL) is a threshold value making it possible to decide (according 
t0 absolute criteria ) whether a student "who knows barely enough" should be passed or failed . 

Calculation of the MPL for a test is not valid unless the number of MCQ 
is more than 30. v 

Use of the MPL involves an ADVANCE judgement (before the test) on the 
relative difficulty of each question and enables a judgement based on 
the test as a whole to be made. 

Calculation of the MPL depends on the collective decision of several 
teachers each of whom has first made an independent judgement. 

2. PROCEDURE 1 

To calculate the Minimum Pass Level (MPL) of a student for a MCQ test: 

2.1 the evaluation board decides what is the correct answer to each MCQ; 

2.2 the board decides which answer or answers must definitely be eliminated by the 
student, other than by chance; 

2.3 the board calculates the acceptability index for each MCQ; 

2.4 the MPL for the test is the sum of the acceptability indices for each MCQ. 
The acceptability index for a MCQ is calculated as follows: 

Carefully study all the choices offered (distractors) and decide which the student "who 
knows barely enough to pass" should be able to reject. For example, if a question offers five 
choices (only one of which is the correct answer) and it is deemed that the student "who knows 
barely enough to pass" should be able to reject one of these choices straightaway, it follows i 
that the marginal student could obtain the correct answer by mere chance approximately one time 
out of four. In this case the acceptability index of the question is 0.25. 

Thus, for a 5 choice MCQ: 

if all the distractors are equivalent, the index 
if one distractor must be eliminated, the index 
if two distractors must be eliminated, the index 
if three distractors must be eliminated, the index 
if four distractors must be eliminated, the index 




- i/5 = 0.20 
= l/4 = 0.25 
= l/3 = 0.33 
= l/2 = 0.50 
= i/l = 1.00 



The procedure described was devised on the basis of an article by L. Nedelsky (absolute 
grading standards for objective tests. Educ. Psy-holog. Meas . , 14 : 3-19, 1954). 
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Let us take two MCQ which are worded identically but where e choice of answers is 
different . 

3 

Question : Which of the following values corresponds to the number of red cells per mm 
of blood in a healthy adult? 

1 2 

A 500 000 A 4 000 000 

B 1 000 000 B 4 500 000 

C 2 000 000 C 4 750 000 

D 3 000 000 D 5 000 O00 

E 5 000 000 E 5 250 000 

In case L, an acceptability index of 1.00 could be considered whil« 
in case 2 it could be 0.25. 

3. COMMENTS 



The MPL has little value if it is not based on a detailed analysis of each of the 
questions in a test, including consideration of incorrect choices just as much as of correct 
answers. 

The validity of the estimate of the MPL also depends on obtaining independent judgements 
from several teachers who have paid careful attention to the educational objectives and the 
level for which the examination is intended. The usefulness of the estimate will be the 
greater the larger the number of teachers involved. 

When the differences between the judgements obtained are relatively small, the extremes 
can serve for defining a "grey zone" below which the results will be regarded as distinctly 
inadequate (failure) and above which the results will clearly indicate a success. For 
example, if the mean of the estimates of one teacher for the MPL of a test is 43% whereas two 
other teachers obtain figures of 45% and 4771., respectively, then it could be recommended that 
any score below 43% should be regarded as a failure, that any score above 47%y&e regarded as 
a success, wnil* ~ score between 43% and 47% should be regarded as being in a grey zone. It 
would remain to be defined what should be done in the latter case. 

If the differences between the judgements obtained by several teachers are large then 
the criteria of the educational objectives should be revised. 




Now \ . . calculate the MPL for all the MCQ you drew up on page 435. 
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SIGNIFICANCE AND FORM OF 
COMMUNICATION OF THE RESULTS 
OF A TEST AS A WHOLE 





RELATIVE 
CRITERIA TESTS 




ABSOLUTE 
CRITERIA TESTS 


1 


BASE 




DISTRIBUTION OF 
THE RESULTS 


EDUCATIONAL OBJECTIVES 


PASS SCORE 




A POSTERIORI 


A PRIORI 

/ 


FORM OF 
RESULTS 


J 


* SCORING 

- ALPHABETICAL 

• - NUMERICAL 


* SCORING 

- ALPHABETICAL 

- NUMERICAL 

* DESCRIPTIVE PROFILE 
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STATISTICAL CONSIDERATIONS 



HOW TO MAKE SURE THAT THE RESULTS HAVE A CERTAIN DEGREE OF VALIDITY 



Content validity is essentially a matter of judgement. Every question should be 
carefully reviewed, together with colleagues, in relation to the corresponding educational 
objectives, to see whether it really measures the objectives chosen. 

Predictive validity is a validation based on the existence of a criterion, of an element 
of comparison which must be correlated with the results obtained. Often there are no 
criteria and in this case validity becomes dubious. However, the customary procedure is to 
make use of statistics to obtain a linear correlation coefficient between the results of the 
criterion and the results of the test. This coefficient can also be derived either from the 
scores obtained (using the Bravais-Pearson coefficient "r") or from a ranking classification 
of the students (using Spearman's coefficient M rho"). In both cases you should refer to a 
book on statistics if you wish to know the formula. 



HOW TO MAKE SURE THAT THE RESULTS HAVE A CERTAIN DEGREE OF RELIABILITY 



To estimate the degree of reliability of the results of a test two series measurement 
must be made under identical conditions and the results then compared. As it is practically 
impossible to work under rigorously identical conditions, various methods are employed. 

Estimation by correlation methods 

Test-retest method : this makes it possible to measure the stability either of the 
procedure for administering the test or of the students, those who have obtained the best 
marks during the test obtain the best matk in the retest. Only the time factor is involved 
in decreasing the reliability coefficient. The procedure is as follows: 

(a) Administer the same test twice to the same group of students after a time interval 
which can vary from a few minutes to several years. 

(b) Calculate the correlation coefficient (r or rho). 

Method of equivalent forms : this measures the uniformity with which two forms of the 
same test, supposed to be equivalent, measure the same behaviours of the student. 

Test-retest method with equivalent forms : this measures both the stability and the 
equivalence of two forms of one and the same test. 

The procedure is as follows: 

(a) Administer two equivalent forms of the same test to the same group with a given 
time interval. 



(b) Calculate the correlation coefficient (r or rho). 



1 If you are allergic to mathematical formulae, pass on directly to page 465. 
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Split-half method : this makes it possible to measure the homogeneity and the internal 
consistency of the test. 

The procedure is as follows: 

(a) Administer the whole test to the same group. 

(b) Divide the administered and corrected test into two halves: 

either by random sampling 

or by upper half and lower half 

or by even and odd numbers (this method is the commonest one). 

(c) Calculate the correlation coefficient (r) of the two halves. 

'(d) Estimate the reliability coefficient for the whole^test by applying the 
Spearman- Brown formula: 

tt 

r = nr 



l+(n-l)r 

tt 

r = reliability coefficient 

n = number of parts into which the t«st is divided, in this case n = 2 

r = Pearson 1 s correlation coefficient 
Kuder-Richardson formula : this measures the internal consistency of the test. 
The procedure is as follows: 

(a) Administer the complete test to the same group. 

(b) Find the number of correct answers. 

(c) Calculate the reliability coefficient by applying the Kuder-Richardson formula 
no. 21 : 

r Lt = _k_ (i - M (k-M) ) 

k ' 1 ks 2 

where 

k = number of questions in the test - 

M = arithmetical mean of the scores obtained by each student 

S = standard deviation of the scores 

Estimation of reliability by calculating the standard error of the measurements : the 
second way of estimating the reliability of a test is by calculating the standard error. As 
we have seen (diagram on page 338) the heterogeneity of the students and the duration of the 
test (speed) are among the factors which affect reliability and consequently validity. These 
two factors are related to the ability or the personality of the partici£ants and are discrimi- 
nating factors which will influence the arithmetical mean of the class (M) and the dispersion 
of the individual scores around that mean (standard deviation) or (S). If the standard 
deviation (S) is low, then the test does not have any great discriminating power so that its 
degree of reliability is low. 
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Since reliability is also related to psychological environment and test procedures, all 
aspects of which cannot be fully controlled, the scores obtained by a student are inc itably 
affected by "non-systematic 11 errors. 

Calculation of the standard error (E) of the scores is one way of estimating the limits 
of the continuum in wnich the student's true score may be situated. 

To estimate the standard error (E) of the scores the following formula is applied: 



E - Sy/ 1-r" 

where 

E = standard error 

S = standard deviation 

r tl 88 reliability coeffic 

In practice it is sufficient w calculate the standard deviation (S) and the reliability 
coefficient (v lt ) and to refer to a table giving the required standard error. 

The degree of reliability is high- if the difference between one test and another is 
low for a given group of students, i.e. E (standard error) is low. 




If you are not yourself interested by statistics and cannot (or do not wish to) call in 
a statistician, the previous pages are only of theoretical interest. 

i 

* *0n the other hand, the next ten pages will help you in making the absolutely essen tial 
minimum of statistical analysis (for relative criterion tests). 
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STEPS ITEM ANALYSIS 

1. AWARD OF A SCORE TO EACH STUDENT 

2. RANKING IN ORDER OF MERIT 

3. IDENTIFICATION OF GROUPS: HIGH AND LOW 

4. CALCULATION OF THE DIFFICULTY INDEX OF A QUESTION 

5. CALCULATION OF THE DISCRIMINATING INDEX OF 
A QUESTION - 

6. CRITICAL EVALUATION OF A QUESTION ENABLING A 
GIVFN QUESTION TO BE RETAINED, REVISED OR REJECTED. 

W 
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1. AWARD OF A SCORE TO EACH STUDENT 



A practical, simple and rap^d method is to perforate on YOUR answer sheet the boxes 
corresponding to the correct answers. By n^acl^g J:he perforated sheet on the 
student f s answer sheet the raw score (number? of correct answers) can be found almost 



automatically. 
ABC 



10 



D E 



12 



13 



16 



17 



18 



19 



20 



A 


B 


C 


D 


1**1 
i ^ ^ i 


1 1 
1 


! 1 > 
1 1 




A 


B 


C 


D 




t 


lx 1 ! 
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x| 
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1 1 

1 1 
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1 1 
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I 
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1 * 


! 1 
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B 
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D 


1* 1 


1 

1 


1 1 


1 
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B 




D 






1*1 
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D 



OS □ □ 



9 

ERIC 



25: 



2. RANKING IN OkDER OF MERIT 



Assuming that the scores of 2i students have been obtained (alphabetical list on the 

left), this step consists merely in ranking (listing) students in order of merit (in relation 

to the score) proceeding from the highest to the lowest score. Let us assume the list as 

under A and then rank the students to obtain distribution 3, ranging from 4 to 27. 

A B 



Student 


Score 




Order 


Studer.t 


Score 


Albert 


7 




I 


Brian 


27 


Alfred 


13 




2 


Frank 


26 


Andrew 


19 




3 


Ann 


25 


Ann 


25 




4 


Emily 


24 


Brian 


27 




5 


Jennifer 


21 


Christine 


19 




6 


Christine 


19 


Elizabeth 


17 




7 


Andrew 


19 


Emily 


24 




8 


E I izabeth 


17 


Fel icity 


16 




9 


Ian 


17 


Frances 


14 




10 


Fred 


17 


Frank 


26 




11 


Felicity 


16 


Fred 


17 




12 


Margaret 


16 


Harriet 


11 




13 


Paul 


16 


Ian 


17 




14 


Philip 


16 


John 


L4 




13 


Frances 


14 


Jennifer 


21 




16 


John 


14 


Margaret 


16 




17 


Alfred 


13 


Michael 


9 




18 


Harriet 


11 


Paul 


16 




19 


Mi chael 


9 


Peter 


•* 




20 


Albert 


7 


Philip 


16 




21 


Peter 


4 
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3. IDENTIFICATION OF HIGH AND LOW GROUPS 



Ebal 1 suggests the formation of "high" and "low" groups comprising only the first 27% 
(high group) and the last 27% (low group) of all the students ranked in order of merit. 

Why 27%? Because 27% gives the best compromise between two desirable but contradictory 

aims : 

1. making both groups as large as possible; 

2. making the two groups as different as possible. 

Truman Kelley showed in 1939 that when each group consists of 27% of the total it can 
be said with the highest degree of certainty that those in the high group are really 
superior (with respect to the quality measured by the test) to those in the low group. 
If a figure of 10% were taken, the difference between the two means of the competence of 
the two groups would be greater but: the groups would be much smaller and there would be less 
certainty regarding their mean level of competence. 

Similarly, if a figure of 50% was taken the two groups would be of maximum size but 
since the basis of our classification is not absolutely accurate, certain students in the 
high group would really belong to the low group, and vice versa . 

While the choice of 27% is the best, it is, however, not really preferable to 25% or 
33%; and if it is preferred to work with \jk or l/3 rather than with the somewhat odd figure 
of 21% there is no great disadvantage in so doing. 

For the rest of our analysis we shall use 33%. 



1 Ebel, R. L. (1963) Measuring educational achievement , Prentice Hall, pp. 348-349. 
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4. 



Calculation of the difficulty index of a question 



Difficulty index 

Index for measuring the easiness or difficulty of a test question, taking ito account 
all the correct answers given by all the participants. 

It is the percentage (7,) of students who have correctly answered a test question, and 

indicates to what extent the students have found the question easy or difficult. 

Calculation 

The following formula is used: 
H + L 

Difficulty index = — - — x 100 

N 

where H = number of correct answers in the high group 
L = number of correct answers in the low group 
N = total number of students in those two groups 
(Do exercise 1, page 471). 



Discrimination index 

An indicator showing how significant iy a question discriminates between "high" and "low 11 
students (see page 468). It varies from -1 to -i. 
Calculation 

The following formula is used: 

Discrimination index = 2 x — - — 

.< 

(Do exercise 2, page 471). 



5. 



Calculation of the discrimination index of a question 
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6. Critical evaluation of a question 



This is based on the indices obtained. 

Difficulty index : the higher this index the easier the question. 

In principle a question with a difficulty index lying between 30'/. and 70% is 
acceptable (in that range the discrimination index has more chances to be high). 

If for a test, you use a group of questions with indices in the range 3 0" 70% 
then the mean index (50-60%) has every chance of being reliable as concerns ,ts internal 
consistency or homogeneity. 

. Discriminatio n index: the higher the index the more a question will dist ingu ish (£o r a 
given group of stude nts) between "high" and "low" students; i.e. it does not favour the 
fow group as compared with the high group. In other words, it helps you to find_out 
who are the "best" students . ^ 

It is an essential index in preparing your question bank. Using the index, you 
can judge questions as follows: 



0.35 and over 


EXCELLENT QUESTION 


0.25 to 0.34 


GOOD QUESTION 


0.15 to 0.24 


: MARGINAL QUESTION - REVISE 


under 0- 15 


POOR QUESTION - HOST LIKELY DISCARD 



1 It does not seem logical; it could be called the "easiness" index but this is not 
the way it is expressed in the literature. 
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CHECK TEST 



Given a group of 21 studencs (see page 467). Using 33% of them to constitute a high 
group of 7 and a low group of 7 (33% of 21), the following table shows the answers given by 
those two groups (high and low) to 10 multiple choice questions (numbered from 1 to 10 in the 
first vertical column). The correct answer for each of those ten question is given 
correspondingly in the second vertical column. In the 14 consecutive columns are shown 
the answers given by each student to each question. 
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D 


B 
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33% 


337* 








HIGH 


GROUP 








LOW 


GROUP 







D 
I 

F. 
I 
N 

D. 



D 
I 

S. 
I 
N 

D. 



r 



4 
7 
4 
2 
0 



11 
14 
7 
7 
3 



3 
0 
-1 

3 
3 



Exercise 




57 
0 
28 
71 
61 



0.57 

0 

0 

0.57 
0.76 




1. Calculate H-L for questions 6 to 10. 

2. Calculate the difficulty index and the discrimination index for questions 1 to 5. 



^Check your results on page 472. 
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CHECK YOUR RESULTS 
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RANKING IN ORDER OF MERIT 
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C 
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E 
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D 
A 
C 
E 
C 
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C C C C C C C 
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t t c 
C E C 
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C C C 
C 
E 
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B B 
C C 



£ 
C 
C 
D 
C 
B 
E 
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E 
C 
A 



B B 
E C 
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E 
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C 
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7 
7 
3 
5 
3 

0 
2 
1 
6 



4 
7 
4 
2 
0 
2 
0 
2 
3 
2 



LI 
14 
7 
7 
3 
8 
0 
4 
10 
8 



3 
0 
-1 

3 
3 

4. 

0 

0 

4 

4 



78 
100 
50 
5u 
21 
57 
0 
28 
71 
57 



Conditions for the application of this procedure for item analyses , in particular; 

1) it applies to relative criteria tests (the procedure leads to a choice of 
questions that lend to maximize variance and ensure discriminatory classification), 

2) it is applicable only to questions scored dichotomously (1,0), 

3) it should not ?>e applied if the total number o£ students is very small 
(a minimum of 20 students could be proposed as a "pragmat ic" criterion). 
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USE OF INDICES 

AIM: Review of Questions 



DIFFICULTY INDEX 



H 4- L X 100 



dif. ind.= 



N 



30 

_L 



50 



60 
J_ 



recommended 



acceptable 



70 



^ DISCRIMINATION INDEX 

2 (H - L) 



dis. ind. = 



l\l 



0 0,05 



0,15 



0,25 



DISCARD 



(most Likely) 



REVISE 



0,35 
I 



GOOD EXCELLENT 
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DOCIMOLOGY 



CORRECTION FOR GUESSING 1 



There is a method for compensating for guessing . . . However, the procedure is far 
from being accepted 1/ all docimologists . A mathematical formula clearly cannot take into 
account personality variations in students. The cautious student will be penalized as 
compared with the student who is more willing to try his luck. 

On the other hahd, (1) if adequate time is allowed to enable all the students who 

should pass to a>\^/ej7 all the questions, and (2) if students are advised to try and answer 

2 

ALL the questions asked then the formula to correct for guessing becomes unneccessary . . . 



. . . However, in case you insist on using the formula, it is as follows: 

number of wrong answers 



Corrected score = number of right answers - 

or CS = R 



.number of possible choices 
W 



N - 1 

For example, if i.n a test of bO multiple-choice questions (one best response type) 
comprising five possible choices, if student X has 44 right answers, his corrected score 
will be: 

CS = 44 - = 44 - ±£ = 40 

3-1 4 



For further details see "Multiple Choice Examinations in Medicine", Hubbarc 7 and 
Clemans, pages 48-51. 
2 

It is rare that it is impossible t*j eliminate at least one distiactor in five entirely 
by chance. In this ca^e trie pi. inability »f answer in^ ALL the questions cjrrectly is already 
increased as compared to blind ch»i*;f. 
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So facilitate the construction of a [question bank it is advisable to enter the 



statistical results for each question on a separate card. 'These cards as a whole will 
constitute the "bank". 

The front and the back of a card of this type could be as follows 



QUESTION ANALYSIS CARD 
FRONT 



Subject 


ENDOCRIN/ SYSTEM 


Nature of question: MCQ 


Objective tested 


Ability to explain the ph>siological functioning of the thyroid gland 


Domain 


COGNITIVE - Level 1 


Question v 


^Wfrich of the following produces an increased secretion of thyroid 
hormone in a normal subject? 


Answers 


A. Administration of TSH 

3. Administration of thiocyanate 

C. Administration of propylthiouracil 

D. Administration of thyroxine 

E. Some other treatment 


Re ference 


Sternberg, Chapter 2, page 112 - prepared by NMD in February 1972 
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QUESTION ANALYSIS CARD 
BACK 



Course 


Date 


Nature 
of 
test 


Group 


Size 

of 
group 


CHOSEN ANSWERS 


Diffi- 
culty 
index 


Discrimi- 
nation 
index 


Conments 


A 


B 


C 


D 


E 


Blank 


2nd year 


6.72 


MCQ 


High 


60 


55 


2 


3 


0 


0 


0 


60 


0.63 




Low 


60 


17 


5 


3 


7 


28 


0 


\ 

CLIN. MED 


6.72 


MCQ 


High 


10 


7 


0 


2 


0 


0 


1 


57 


0.30 




Low 


10 


4 


0 


1 


1 


4 


0 


2nd year 


6.73 


MCQ 


High 


62 


56 


2 


3 


1 


0 


0 


60 


0.61 




Low 


62 


18 


8 


3 


8 


24 


1 


2nd year 


6.74 


MCQ 


High 


/0 


60 


0 


2 


3 


5 


0 


59 


0.57 




Low 


70 


20 


7 


4 


2 


30 


6 


Specialty 
Board 


7.74 


MCQ 


High 


20 


18 


0 


2 


0 


0 


0 


80 


0.20 




Low 


20 


14 


0 


0 


1 


5 


0 
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CONVERSION CHART 



In the course of your reading you will come across research results expressed 
in cumulative percentages, percentile equivalents, Z scores, etc. . . 

This conversion chart will help you to make comparisons, wtfen necessary. 
However, it cannot be used to equate scores on one test to scores on another test. 




\ 



EXERCISE ) (Check your answers on page 484) 




Question 2 



Question 3 



So-called "objective 11 written examinations are not suitable for measuring 
on? of the following. Which ? 

A. Ability to recall knowledge precisely. 

B. Ability to solve problems. 

C. Ability to make decisions. 

D. Ability to communicate with the patient. 

E. Ability to interpret data. 



All the following stages, EXCEPT ONE, are recommended 
the "essay" type: 



r scoring tests of 



A. Write the elements of the answer for each of the questions asked. 

B. Correct the answers question by question rather than student by student. 

C. Determine the pass score on the basis of a sample of answers. 

D. Correct the answers while preserving the anonymity of the students. 

E. Identify three levels only: honour, pass, fail. 

The content validity of a written test is usually obtained by means of: 



A. 
B. 
C. 
D. 
E . 



Collective and careful review of the questions. 
Pearson's corre; .cion coefficient. 
Factor analysis 

An "inter-rater" reliability coefficient. 



a/to 



meafi discrimination index. 




\ test with 50 question is administered to a group of 45 students. There is 
a choice of five answers to every question. Only one of these choices is the correct answer. 
One point per correct answer is allocated in calculating the total score. 



0 



Assuming that none rt f the students have any knowledge of the test subject (i.e. they 
choose their answers by guessing), which of the following will be closest tu the mean 
score of the group? 



A. 
B. 
C. 
D. 
E. 



0 
5 
10 
15 
25 
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On dividing this group of 45 students into 3 groups of 15 each, on the basis of the total 
score of each student, it is found that, ior the first question, nine students in the 
high group and three in the low group have given the right answer. For this question 
the difficulty index is: 



0 



A 


1 ?7 

1£ /o 


B. 


27% 


c. 


30% 


D. 


40% 


E. 


60% 



Under the same conditions, the i*iscr imitation index is: 



A. 


0. 


12 


B. 


0. 


27 


C. 


0. 


30 


D. 


0. 


40 


E. 


0 


60 



Questions 7 and 8 



The following data concern a multiple choice question to 300 students, the correct 
answer being D. 



0 




8., 





Cuoice o? answers 


A 


B 


C 


D 


E 


No answer 


High group (100) 


22 


1 


10 


67 


0 


0 


Low group (100) 


46 


5 


16 


33 


0 


0 



A. 
B. 
C. 
D. 



These data show that: 

all the distractors were of good quality; 
half the students answered the question correctly; 
the question was of high validity; 
The question was not very relevant. 

In view o£ these data, the examination board may decide: 

A. that this question should be reviewed s,.ce it is insufficiently discriminatory; 

B. that this question should be discarded from the question bank; 

C. that this question is of low validity; 

D. none of the above. 
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Question 9 



What could generally be expected on doubling the length of a test whose mean 
discrimination index is 0.52 (by adding questions more or less equivalent to 
the previous ones)? 



A. 
B. 
C. 
D . 
E. 



Only a certain increase in the reliability of the test. 

A certain increase in the reliability and the validity of the test. 

Only a certain increase in the validity of the test. 

A certain decrease in the reliability and validity of the test. 

No effect on either the reliability or the validity of the test. 



Questions 10 to 15 

Use the following key in asnwering this series of six matching type questions: 

A = traditional oral test 
B = written test of the essay type 
C = so-called written "objective" test (MCQ) 

D = standardized practical test, or written and oral simulation tests 
(programmed examination) 

Indicate the type of test most suitable for evaluating each of the following 
performances : 



Question 16 



10. 
11. 
12. 
13. 
14. 
15. 



Recall of concepts. 
Ability, to solve problems. 

Ability to communicate satisfactorily with the patient. 

Verbal expression. 

Skill in examining the patient. 

Ability to make a synthesis. 



The system of "relative" criteria of competence implies the following 
consequences, EXCEPT ONE. Which? 



A. 

B. 
C. 

D. 

E* 



Leads to an embarrassing disagreement among those responsible for applying 
the resultant decisions. 

Leads to the failure of certain students in a particularly competent group. 

Enables one group to become the arbiter of the standards according to 
which it is judged. 

Enables "low group" students, who are however superior to the mean of the 
whole group to which they belong, to pass. 

Creates an arbitrary fluctuation in the desirable level of competence at 
a given moment. 



\ 
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Question 17 



Question 18 



A Z score corresponding to the mean (of a nonral distribution). is equal to: 

A. -1.0 

B. 0.0 

C. +1.0 

D. A value which cannot be determined from th'^ preceding information. 

The author of the following question wa* a r /Ked to establish its minimum pass 
level (MPL): 

The diameter of a normal erythrocyte (according to WINTR03E) is equal to 



(in 




A. 


4.5 


B. 


6.5 


C. 


7.5 


D. 


8.5 


£. 


10.5 


He 


felt 



reject right away choices A and E. 

Indicate which among the following minimum pass levels corresponds to this 



choice: 




A- 0. 


10 


B. 0. 


20 


C. 0. 


25 


D. 0. 


33 


E. 0. 


50 



Questions 19 and 20 



19. The calculation of the discrimination index provides statistical data which ~an be used 
in absolute criteria tests (true or false). 

20. The calculation of the minimum pass level (MPL) can be used in criterion reference 
tests (true or false). 



* 
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Answers suggested for the exercise on pages 479 to 482 



Ques tion 


1 

Suggested 
answer 


If you do not find the correct answer, 
consult the following pages again 


1 


D 


417-418 


2 


C 


420 


3 


A 


461 


4 


C 


sirn:/2e 'Yule of y 


5 


D 


469 


6 


D 


469 


7 


B 


469 


8 


D 


470 


9 


B 


Vc 


10 


C 






11 


D 






12 


D 




324, 416 to 429 


13 


A 






14 


D 






15 


B 


j 




16 


A 


457 


17 


B 


478 


18 


D 


458-459 


19 


False 


472 


20 


True 


458-459 
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PLEASE 



IF YOU KNOW HOW TO DEFINE SPECIFIC OBJECTIVES 

IF YOUR COLLEAGUES TURN GREEN WITH ENVY ON 

READING YOUR CRITERIA 

IF YOU ARE ABLE TO CHOOSE THE MOS" SUITABLE 

TEACHING TECHNIQUE 

IF YOU CAN PUT IT INTO PRACTICE 

IF YOU AREA LEADER 

IF YOUR STUDENTS ADMIRE YOU 

IF YOUR EXAMINATIONS ARE VALID 

IF YOUR SCORES ARE OBJECTIVE 

DO NOT BE INFLUENCED BY ALL THIS . . J 

. ..AND OVER-ESTIMATE THE IMPORTANCE OF YOUR OWN SUBJECT. 

f 

DO NOT FORGET RELEVANCE . . ! 

THE RELATIONSHIP BETWEEN YOUR TEACHING AND THE 

INSTITUTIONAL OBJECTIVES. 

ERIC 
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PURPOSE OF THE CHAPTER 

The purpose of this chapter is to facilitate the task of anyone who wishes 
to prepare and run an educational wcrkshop. 1 It puts into practice most of the 
theoretical concepts mentioned in the Handbook. 

It contains the basic documents (or refers to the documents) required to 
organize a very short workshop (lasting 3 to 4 days) known as a MINI-workshop * 

It also includes simple instructions on how to use these documents and 
describes what has to be done to get the workshop under way # 

The chapter obviously makes no claim to qualify the reader to organize all 
possible types of educational workshop regardless of the educational philosophy 
involved. The type of workshop proposed is oesi gned to use the WHO EDUCATIONAL 
HANDBOOK (1976 or 1977 edition) as its source of theory. 

Nevertheless, most of the general principles involved are also valid for 
longer workshops* 

This chapter responds to the need often expressed by those attending 
workshops run by WHO to pass on a knowledge of the systematic approach to 
educational problems to their colleagues by means of a short workshop* 
The method proposed itself uses the systematic approach * It also relies on such 
educational principles as: 

* Allowing the participant to prepare and select the objectives to be 

reached will increase his motivation; 

* Giving the participant an active role will make teaching more effective; 

* Providing the participant with regular opportunities to see the progress he 

is making will increase his learning speed and improve the .quality of the 
knowledge and skills he acquires* 

For a description of what is implied by the term "workshop" see the next page* 
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WHAT THE TERM WORKSHOP IMPLIES 




A workshop is a meeting during which experienced people in responsible positions 
come together with experts and consultants to find solutions to problems that have 
cropped up in the course of their work and that they have had difficulty in dealing 
with on their own. Participants themselves select the objectives they wish to reach 
and help in choosing the problems for group work.*" 

AN ESSENTIAL FEATURE OP THE WORKSHOP IS COMPLETE ACTIVE INVOLVEMENT EY EACH 
PARTICIPANT: THE WHOLE POINT OP ATTENDANCE 13 TC WORK AND TO LEARN FRC*1 PRACTICAL 
EXPERIENCE 

One of the commonest methods used in workshops is group discussion of selected 
problems, the size of the group being small enough to encourage full participation by 
each member and large enough fcr each member to gain from the experience of tne otners. 
There is nothing magical about a small group but it does offer each member aa 
opportunity to make his own contribution. It gives participants as far as possible 
a chance to discuss, and solve, those problems of greatest interest to them. The 
fact that each member can find something in what the others have experienced that 
has a bearing on the questions of most interest to himself will make his work more 
meaningful. The workshop method makes everyone (organizers and participants) 
responsible for helping to find solutions to tne problems selected. Participants 
may have to act as group leaders or rapporteurs. The organizers are generally 
there to be consulted by participants and to help them where necessary (but they 
do not give lectures cr impose solutions). 

The workshop programme makes provision for plenary sessions, discussions in 
small groups and other activities but does not follow a strict hour-by-hour timetable. 
On the contrary, the programme adapts itself to the way the work is going. For 
example, a plenary session will be held whenever there is a need to pool the results 
of group discussion, to clarify a point for all participants or to introduce some 
activity that requires unanimous approval. 

A MINI-wcrkshop Is a short workshop (lasting three or four days in the case 
described in this document). 

These problems and objectives are occasionally decided (in most cases by 
correspondence) before the workshop begins to allow the various participants 
time to prepare for it. However, this does not happen with a mini-workshop. 
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AIMS OP A MINI EDUCATIONAL WORKSHOP 



The workshop aims at Introducing participants to a systematic approach to 
educational problems. They must, so to speak, not only be made "hungry for more" 
(i.e. anxious to carry on learning about the subject and increasing their skills) 
but also given "food for the Journey" (Documentation containing enough references 
to enable them to make progress after the workshop is over). 



The workshop aims at stimulating a given proportion of participants to 
wish to reach at least those objectives set out in the Educational Handbook 
(see pages 525 to 527 of the present chapter) in the course of the ensuing year. 



It has been founu from experience that some participants may go far beyond 
these aims and embark on activities such as: 

- defining X% of the specific objectives ior the subject they are 
responsible for; 

_ replacing X* of traditional lecturing by a more suitable method; 

- starting a bank of examination questions that meet the criteria of 
objectivity, validity, etc. (X questions); 

- calculating the 3 iscrimination index or acceptable level of performance 
for X56 of the examination questions; 

etc. etc. 
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HOW TO PLAN A MINI-WOKKSHOP 



You have already had first hand experience, as a participant, of an 
Educational Workshop and you have decided to organize and run a mini-workshop 
yourself in order to let your colleagues know about the systematic approach 
to educational problems. This will, however , involve a good deal of work. 
The following CHECK LIST has been drawn up to help you. 

Only those items considered ESSENTIAL appear on this list. Not all of 
Liem may be applicable in your case and you may also find that some items 
you need are missing. 



Unless you start work AT LEAST six months beforehand you will be increasing 
your chances of failure. 

Tne greater the flexioility ana" adaptability of equipment and staff the higher 
the chances of success. 

One of the aim^ of the workshop is to meet the needs of the participants. 
Your apologies will be n,> use to them if something goes wrong. On the other 
hand, they will be favourabl) impressed if, when s ^thmg unforeseen does happen, 
corrective measjres are Liken lo keep tne workshop running smooth l>. 




evertheless 



THE SUCCESS OF A WORKSHOP WILL DEPEND LARGELY 
ON THE WAY IT IS PLANNED AND ON THE ARRANGEMENTS MADi 




THE OPENING SESSION 



r 4 it i *_ r *p. it 



' lin h uiit\p«.<ted will ilwa.& happen.' 




ERIC 



506 



WORKSHOP 



ACTION CHECK LIST 



No. of days before 
or after workshop 



see page 



B 
E 

F 
0 
R 



A 
P 
T 
E 
R 



Decision to organize a mini-workshop 
Open a file 

Find a source of funds 
Have a draft budget approved 
365 to I - Set the dates for the workshop 
- 180 I - Choose the place to hold the workshop 

Book a meeting room and accommodation for 
the participants 

Define the criteria for selecting participants 
Appoint a Committee of Sponsors 
Choose the assistant-organizers 
Take account of the working language 



- 120 



60 
30 



- 8 

- 2 



♦ 10 

♦ 15 

♦ 30 



Start the procedure for inviting participants 
(inform them of the aims of the workshop ana 
send them Doc. 1 (Working methods) and Doc. 2 
(Theme of the workshop) 



Select the participants from tnose applying 
Inform participants that they have been 
selected and 

Send them the documentation (Educational 
Handbook and Document 3) 

Arrange for document reproduction equipment 
to be available 

Prepare a checklist of the equipment required 
Inform the press 



Review the list of participants 

Arrange the room and inspect the premises 

(with equipment check list) 

Call a meeting of the assistant organizers 

and review the programme for the workshop 

Have .a friendly drink 



Background organization 
of the workshop 



timetable of work 
functioning of the 
workshop 

- organizing the breaks 

- group photograph 

- make an immediate 
evaluation (Doc. 4) 

Send , a leTter of thanks to the assistant organizers 
Prepare a report on the workshop 



* 180 to 



Send the report to the participants 

to the responsible authorities 
Start long-term evaluation 

- collect data 

- visit the participants 

- organize an evaluation meeting 

- publish an evaluation report 



505 
507 
507 
507 
507 
507 

508 
508 
509 
509 
510 

510 



521 
521 

523 

531 

531 
531 

532 
532 

534 

534 

534 
535 

536 

547 

537 

547 
547 
547 

547 
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OPEN A FILE 



Correspondence relating to the planning, running and evaluation of a workshop 
will soon reach proportions that call for proper filing. A suitable system might 
be a looseleaf file with the following subdivisions: 

- Budget 

- Workshop site 

- Selection of participants 

- Selection of assistant organizers 

- Documentation 

- Equipment checklist 

- Publicity, press, etc. 

- Evaluation / 



FUNDS 



Whatever social and political system you are forking under you will need a 
budget. The person or persons who will ultimately be responsible for authorizing 
the expenditure involved will need at least one ESTIMATE. To work this out the 
following simple formula is suggested: 



E - (T * S)N x 1.25 



E » Estimate 

T = Cost of return travel 

S s Living expenses (accommodation, food) 

N = Number of participants 



per participant 



In other words, travel costs and living expenses will amount to 8o# of total 
costs, leaving the remaining 20J6 to cover the other expenses (room, reproduction of 
documents, etc). Where necessary add a percentage corresponding to the annual 
rate of inflation and get your budget approved . 



DATE OF THE WORKSHOP 



As the workshop will nearly always be attended by teachers, this should be 
taken into account when setting the dates of the workshop in order to avoid 
clashing with their professional commitments (e.g. sitting on an examination board, 
annual congress). 

You should also check whether the dates coincide with public or religious 
holidays, sports events or political meetings, as these may create problems with 
regard to reserving hotel rooms. 



CHOOSING THE PLACE FOR THE WORKSHOP j (over 6 months before D day) 



It has been found from experience that, to make sure participants will attend 
on a full-time basls r the workshop will have tc be held in a place far enough away 
from where participants live to enable them to take part in all activities without 
interruption and prevent them from being able to go home after the sessions or, 
more importantly, return to their laboratories or their patients. 

♦ This obviously implies a substantial financial investment but one that is Justi- 
fiable from the point of view of cobt/ef fectiveness. 



1 Before working this out, rea«i the paragraph on "choosing the place for the 
workshop" on this page. 
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The place chosen should be agreeable and the conditions comfortable enough 
for participants to be able to recall with pleasure their first full-time plunge 
into the depths of the systematic approach to education. 

BOOKING A MEETING ROOM AND HOTEL ACCOMMODATION J (6 months before D day) 



i 



Waste no time in making the necessary bookings. Do it in writing and insist 
on written confirmation detailing the conditions you h^ve specified, particularly 
as regards to meeting room (see p. 532 for details)* Wherever possible, a visit tc 
the premises is recommended before making a final decision . If the meeting roctn 
is too small, too noisy, badly ventilated or badly lit, the workshop may suffer 
irreparably. 



CRITERIA FOR SELECTION OF PARTICIPANTS 



Number of participants 

It has been found from experience that as many as 35 participants (seven groups 
of five) can be handled by one organizer. By following the maxim, "the less you 
teach the more they learn ", it should be possible to increase the number. of participants 
still further, 

1 

However, there are no strict rules and it is advisable not to haj/e more than 

about 15 participants for a first trial. j 

i 

The number of participants is in practice limited by what the organizer feels he 
is capable of coping with and by wnat the participants think they need in the way -f 
help from the organizer. 

The documentation provided for the mini-workshop should be such as to a^low 
each participant to advance by his own efforts and with the help of the stimulation 
provided by other participants rather than with -che help of the organizer . 

Type of participant 

* Homogeneity - 

Although \ is not necessary to have all the same kind of participants for a 
mini-workshop, since this type of workshop is mainly intended to provide a stimulus 
to the individual , it is advisable to select participants who will be in a position 
to work together after the workshop is over and who, by forming a critical mass , will 
have a greater chance of success. 

On the other hand, it has been found from experience that it is not recommended 
to have a workshop ^designed to act as an INTRODUCTION to a new approach ^attended by 
a mixed group representing several professions (for example: physicians, nurses, 
dentists, medical assistants, sanitary engineers, etc.). The mini -educational 
worKshop is going to force each participant to question many concepts that had 
previously seemed firmly established. This process is hard enough without 
increasing frustration by insisting that it is carried out outside the peer group. 
The aim at this stage is not to create team spirit. It may be necessary to 
organize a mixed group later on in order to reach this goal. 
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* Voluntary participation and willingness to innovate 

Attendance at the workshop should be VOLUNTARY and each participant should 
already have demonstrated his desire fur change by having adopted new methods 
of his own. He should if possible be in a position of responsibility or be 
likely to acquire responsibility. 

In cases where some of the documentation is not ye* available in the national 
language, or if assistant organisers who do not speak the national language nave 
to be called on, allowance must be made for this and participants selected who ^an 
at least READ the language used in the documents available. 



To be accepted as a participant it is ABSOLUTELY ESSEETCMAL that : 

applicants are aware they will be FULL-TIME PARTICIPANTS for the 
duration of the workshop and underta ke to abide by this provision . 



COMMITTEE OP SPONSORS 



Setting up a Committee of Sponsors will not only permit you to honour 
influential members of official circles but will also <1raw the attention of 
such circules to the action you nave initiated and encourage them to follow 
it up. It is important that people in administrative positions (such as 
Pectors, Directors of Health and Deans) should be represented on such committees 
which will be called on to apply the selection criteria defined earlier. 

SELECTION OF ASSISTANT ORGANIZERS (4 months before D day) 



As you yourself have already attended a workshop of tnis Kind, do not oe 
afraid in your effort to take SOLE charge of a group of about 1^ participants. 

However, if you have not enough confidence in you °lf yet call in 1 a more 
experienced consultant, preferably from another scnool faculty , and this will 
allow you to^take a rather larger group (20-25). 

In subsequent el forts with greater numoers of participants you .are strongly 
advised to take on two (for 20 participants) or three (for jo participants) 
assistant organizers (from those attending an earlier workshop). You will have 
to make sure at least four months before the workshop stirts that they will be 
able to come so as to give them time to make further study of the documents, in 
particular of the WHO EDUCATIONAL HANDBOOK . 

You are recommended to choose assistant *jrgani/,ers belonging to disciplines 
other than the ,e represented by the participants in order to prevent the proceedings 
turning into a discussion of their subject by experts rather than a consideration of 
the methodology pf education. 

The assistant organizers will have the task of finding answers to questions put 
forward by the participants and of channelling any questions from participants thafr 
they ->:,not deal with themselves to the principal organizer. 



By contacting either your WHO Regional Office or the Chier, Educational 
Planning, WHO Headquarters. 
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WORKING LANGUAGE 



Apart from the WHO Educational H-nJbocK, the remaining documents (i, 2, 5 and 4) 
are short enough to allow their translation into the language of the participants of 
the worKshop. At any rate all discussions, whether in small groups or in plenary 
session (unless outside consultants are used) may obviously be carried out in the 
national language . Make arrangements at once for the necessary translation to be 
done. 



INVITATION TO THE PARTICIPANTS 



(4 months before D day) 



It is now time to start the procedure leading up *o the final selection of 
participants (see page 52l). As far as possible, you are recommended to generate 
a demand for places in excess of the number you have decided on (page 508) so that 
you will be able to correct the situation if the~e are any last minute cancellations. 
You are therefore advised to get in tough now with the colleagues you wish to contact. 
A personal letter will generally be preferable to posting up a notice, but your 
decision here wLll depend on local conditions. What "information" should be sent 
out at this stage? The main points to be covered are: 

• a) aims of the workshop - on the basis of the contents of page 504 
prepare a letter of invitation adapted to the target population 

(b) v»h?t is implied by the term workshop - you may annex to your letter 
the text on page 503 

(c) working methods of the workshop (document No. 1) 

(d) theme of the workshop (document No. 2) 

The text of the last two aocuments (1 and 2) will be found on the pages that 
follow. You may have the requisite number of copies made by means of a stencil 
if you do not have access to other methods of reproduction such as an electronic- 
stencil or a fast photocopying machine for which the pages of this document may be 
used as originals. In any case the pages should be renumbered and the place and 
dates of the worKshop should be mentioned on the cover page of each document under 
the title "workshop in educational planning". 

Your letter of invitation should also mention that full-time participation 
is essential (see the text in the box on page 5 U y) and indicate any language 
stipulations. 

Lastly, you should set a deadline for applications (^5 days after the date of 
dispatch of the letter of invitatior) , mentioning that each successful applicant 
will be informed of his selection at the latest ^5 days before the start of the 
workshop and that the basic decuroento 2 will be sent to him at that time (under 
separate cover if you have a Hmited budget). 



Which exists in In^lish. hr»'n«h. Italian and .Spanish and 15 in preparation m 
Bulgarian. Czech, dcrmaii. Hungarian, Polish and Portuguese. 

2 

N'.B. Now is the time t > ,rdor what mpios yu need of the KI)U( AI lONAi 
HANDBOOK FOR HKALI H PLKSU X M L . WHO t<)77. fr,m your bookseller (see list on back 
cover) unless your national authorities already have a stork which they can make 
available to you. 
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WORKING METHODS OF THE WORKSHOP 



27 1) 



Work trig rrethcds 



The wcrKing methods offered to ycu i' r r this reeting may re somewhat 
different from those you are traditionally used to. It does not mean that 
these are new methods. Tr.e^ nave oeen widely used outside the University. 
Their efficiency has been experimentally tested. They derive from the appli- 
cation cf recent education theories. If tr.ey were to be identified by their 
main characteristic one would say the> put empnasis on active participation of 
the learner while ' n traditional systems he is maintained in a relatively more 
i^ssive role. The aim of tnose methods is also to encourage you to demonstrate 
a critical and constructive attitude and to Incite you to find out, solutions 
to yc jr own. proolems. To that effect practical exercises will 1 *ad you to 
propose realistic solut^ns. Generally speaking, the working methods used in 
\i *■ worksncp wi*l er.ace you t^ put tr.e educational principles recommended in 
i!.e EDUCATIONAL HANDBOOK ' pra«:ii*_". 

T* c wcrv.inf retn'df "irv* V e !cll'Wi:iff: 

• f* ' r " 1 1 e ! it ore c * *i 1 i r, e c- t 1 \ or. 



- I" ^ racr *:~ '^r.ro *.:.at t:<t ~* . ' y meets your educational needs 
y . w*»l ^ v : " r>'i tc sele.v, \: *=• ' ; t iues yc u wi sr. tc reach before 

*■ e er d 'i V ** a r^c T ( dV ure: t N - » /) . 

: -e . >.ra rv r**a i : * i: : 1 ^rr ~^r ,*r, Orientated tc wards practical exercises) 

- I" srier t lac le t „. - - a zj-ise rf thee ret leal knowledge in the 
• ■ "(3 .-,atl ra rlanr. Ing trat you ray need to find solutions t<> 
:ra't a. rr.r.<rc f r.*i '■tta r Ve -cj^ctives yu ::avo se. etc 5 , y-u will 
.♦j : . 1 V J *„ r c n :.r»' ~ f "! p -t/'d J 1 «*-'*\tr# . It should therefore made 

/•sr v„ : "'t t'^'. v^re v:"!. :,e ; lecturing ' n the part cf t*.e 

r^ar.^eri, t evr— a V,' ~rt !ntr d t r n t 



T* r ?*• r»T. *: :*e ■" . : , r. ^'en. , ial iy ac t: ^ f 'rsl wrr.<lng serslcr*. 

7"" r r .* «■ . -r- "i*. * r^T'*- acs A gr "-e* tc r ave oeen ur.d'*rst<"xi 

.:. V o /a : t- <■» " r " \ . ' ov r * r t Intended n< wever t " he 

; r -3'-; v i r ^ \ r«- v ■ ; «» a ' * . * e ■ e ^ o! tr <*r a .r!;.^ wr rv gr' :f 



^ o t *t'*+' • *> r «r«. p ; » . *r t* e f.d * at" nn. ,"ar/it'~<' V . 

'Tf •# . ' . t r d ; >i><i*.j * ' *y -\ . Tier disc *ss i^r, will ctart 
r - ' -:ar. f » ■ 'r ;r ;'Co<i : utl rr. T'Js excr.ange ^f views ray 
- - r . i ^ t re v tor 1 " * ■ t y t e o*.' ! re .'^a i i ?v '-'^p ; if t:,o 
^/or- o'* 't. le i* '''j* 1 • ' • *» r ^r- u; Ms a specific tasK. related 

t' t ' r - , :• : 4 * w.S?^. 'I n ^ls task 



: t*"j r / 'r i 'vlnxat* n ar 



5. Group presentation 



- Plenary sessions wiii be nela as cfter. as necessary. Tneir aim will 
be to allow group solutions tc oe presented, not for the purpose of 
judging groups but to compare ideas for possible consensus under the 
leadership of either a participant or an cryar.izer. 

6. Next working day preview 

- Each day before closing a short per'od *i 11 do reserved for a preview 
of the following day's activities and a reminder cf tne airs of reading 
assignments. 

7 . Individual consultations 



- In view-cf the blc_ epical rrir'iple r : ^dividual di: :erences i o is 
rost iikei;, toat part:c:rar*r wi.l prepress at. dif:erer.t rates, desiring 
to study in r,ore or }ps.s aer-tfj -'Ortair: -uestiens, or be interested ir 
differing appl i' atiens o: t r e tr.ezries arc retrede propesed during the 
works? 'p. '?' mat a: 7 c rran : ;* A rr wl . I rra.^.e tne^se ; vc*r; a va ikio. e / r 
rec ue t ' r ' r 'i i v i dua I o : . ~ - 1 tr. t ; : r. c on su d," ec ts c f tua . ! r. :,e re:, t . 

Fc rra t i ve e va \ ua t : r. 



. 1 Pretest - Ir crier tc czlr r - t vc,r e::crts and in: err yc~. t y'-r 
pr.gress i r \ ^ar r for uu^ir.* *••• rnrr r, a*" evaluation prccess is 
pre y.sr.t : to y j. Leta,.r, e ✓ iver. n pare * tr.e Educatl^ra. Ka: db<" ' 

7olr te.ot w;.l »- r , . [ aer-.*./. t y tr.e "eaucatior.a i area" :r 
whicr. a re: :*er:.er ra/ r.eeiea. IV e os'ect or t:.~ protest is not f T:r : 
tne 'rlgr.t" answer tJt vir\ .y t: renter tr.at you nave net round it . The 
purpose cl tr.e v r.".cr : .s t n^lc ycu v find it, either during the wrrk- 
snon itself cr In tne er: /r/ r^ntr.s. Tr.e pert -test will rive >ou an 
eppcrtunity ac tire *'es ry : reas-Klv y s ur pr^TOT.. 

Daily evaluation - At v p r nd " : ea~r daily session tiTe will be set 
asiae i^r ,^int asserr.rr^nt : 're ex tor i to w;.; y * he wcr^irg metroos 'isen 
nave no I red you towards re/- -.Irr tne '0*eetlves you nave sele^t^d and : ^r 
prop-sirv any o r ar.^ ir • ^r t.or ir? ' : tne wirKSrcp . 

^ . 7 Eva 1 ua t ! c r vjos t . *■ nna re - C": ' r *> . y so fore tr e end " f t? e root i r.r vo* . 
wi 1 A do i r.v i v-d t' e> press y .r <. r . : '. ' n ^ n or e " rrar : za t i ^ r. ' tr e 
workshop oy .T^^ r r : ^ quest rrra!r** (l)< ojrrent ) . 7r.<* rer^rr wi ll n^ 
ana! y zoo d .rlnr *re . ast sessi'n. 

b.t* Lonr-terr ova.uati'n - It 'rnr-r V ro.i in asseminr V.o renef it 

you will a ■ t'/P fr'ir 4, ir w ivnh^p. wl., t>e asked *,< ae? ire explicit./ 

the p r' f p . p i ^ na . ' ; p' M vr-r / ^ . expect to rea rh wi t l i n the nt- x t two i ve 
rr>c ntr.s I r re * a t * • r t' tn° ex ; or i en r e a i I rod dur ' r^r tr e v rks^ cp. A 1 . 
parti oi parts are re' Tfp? J^d t reot t /^tner twelve r^ntnr a J tor tre 
workshop Ir f nior t* ova' ,ate wrat tney rave acnipved. 

Those various aspe^ti rf tr.e w-rrlr** Teth * l at e used jurln^ tr.e 

worksncp w< ] ] j>e car!' ! p J j .r!r.».* t* e : 4 rr: sesr:T. 





You may now start preparing for the workshop in 
whatever time you have available. 

-If you want to get a general idea of the field that 
will be explored during the workshop, study Document 
No. 2 f Introduction to Educational Planning) 

- If you want to select the personal objectives 
you wish to reach by the end of the workshop, 
study Document No. 3 (Programme Preparation for 
the Workshop , which you will be sent if selected 
to participate 

- If you want to go still further, you »jJL .nake use 
of the Educational Handbook in the light of the 

</ jectives >v>u have selected. 
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THEME Or TrlE WORKSHOP 



The pattern of teaching has remained the same for centuries. The 
privileged enclaves of the universities have often remained indifferent 
to agitation outside them and the needs of the community, the practical 
aspect of the matter, has been left to chance. The details of each country's 
situation are changing at an ever increasing pace but unfortunately no 
attention or very little attention has yet been paid to them and the training 
of health workers still follows the traditional systems of the past. 
There is a need to ensure that teaching programmes are relevant . 

To continue to copy past models, or foreign models in the case of 
developing countries, is out of the question. 

The aims of the training system for redlcal and health workers , at all 
levels, must be reviewed in terms of tne needs of each country and must proceed 
from prospective study . 

No educational system can be effective if its aims have not been clearly 
formulated. The members of a healtn team must be specially trained for the 
tasks they will be called upon to perform in view of the conditions prevailing 
in the services in. which they will be expec**H to work. 

These tasks must be defined in relation to a plan which lays down a 
health policy and specifies the type of services to be provided, the 
priorities to oe observed, the number of staff required, etc. 

This will make it possible to adapt professional education tc the needs 
specified in this way. 

A stcck cf rar.agement, plar.nir.r; arc educational skills is available to 
nelp teaching staff at existing or future health science centres and to assist 
health administrators who r.ave training responsibilities. 

The definition cf educational objectives for educator ard training centres 
is essential as a first stage. 

Trus a training rrogramme, instead cf deriving from a fund cf knowledge 
built up r,cn-select;vely r ver tne centuries , should be selectively designed in 
terms o f tne educaticnal airr to se reached . If tre aim changes with time, tne 
programme snculd als r be arended ir. consequence. 

Educaticnal ct. cctiv^s rrou.d tr»-r^fcre take their sour-e fr r a study cf 
needs and possibilities a*-. *-* r *d be nased cn a definition *-? the tasks that 
staff will oe called n 4 ' r i\ during their new. ins: life ir. any ^iven t>pe 
of he a 1 1 h wo r k . 

Preliminary de* initio "i tre tas.'.s G.eaitr. ^are) to be provided is stressed 
oecause it is a prerequisite tor any relevant definition of educational r bjecti ves 
(see diagram on n^xt page). If t,ne bases from wrich the educational obje^iv^s 
are derived are cao , u.er. tre educational system used, norfever "good" it is, is 
likely tc give "'bad" resjlts, aid a "bad"' messa^ in likely to bo c mmun ! rated 
"well" , wuich is by no means tne irtenaed air*. 

The method trad: ti^ ndi » % used draw up an ed- cati^nal programme if t' bring 
together eminent professors and tne result cf their delioerationr is presented as 
a list of chapter headings. Often ev<st:np; i r-vrafrmer, are used as tne main 
source of data f>r tre prepar** r. / tne new p>- t-rirrrre . Tr;<* pr<_f enccrn 
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ORGANIZATIONAL DIAGRAM SHOWING THE RELATIONSHIPS 
BETWEEN THE SUBSYSTEM "EDUCATION" 

(inside the broken-line quadrilateral) 
AND THE SUBSYSTEM "HEALTH SERVICE" 
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indicate the number of hours to be devoted to the * arious subjects tc be dealt 
with: this generally leads to a conflict of personalities and it is the most 
forceful, the most persuasive, sometimes the most irascible or noisiest of the 
participants in the discussions who obtains the largest number of hours- The 
result is that tne time variable becomes a constant and that competence becoires 
a variable . 1 

The general objectives of a medical school, for example, are, some say, 
axiomltic: "We train doctors of international quality. It is not necessary to 
develop the description any further; medicine is universal." However, when we 
try to get ?he teachers to define a little more fully what they are talking about, 
we see how wide and fundamental the divergencies are as soon as we leave the 
sphere of generalities. The conflicts between fundamentalists and clinicians, 
between advocates of preventive and of curative medicine, are the* results of those 
divergencies. This conflict becomes acute during the periodic curriculum reforms. 

It is proposed to define educational goals by means of behavioural terms 
corresponding to the tasks to be accomplished. In other words, the definition 
must indicate wh at the graduates of a given school will be able to DO at the end 
of their period of education or training that they were not able to d o frgfore. 

These goals define what the student should be able tc do, not the teacher. ^ . 
A teacher's expectation of high student learning motivation will not become a 
reality until students feel a freedom to learn, that is to say when they become 
involved in tne decisions concerning their own educational objectives. Moreover, 
these objectives have to be explicitly defined and o.Tered for their choice; even 
better, the students will have participated in establishing them . 

When educational objectives are described in a practical and detailed manner 
it is then possible to determine with some precision those learning a ctivities 
suitable for attaining an objective and those which are almost or completely unsuitable 
This is the second stage. Now is the time to work out the study programme (the 
sequence of learning and teaching activities and their coordination), to estimate 
equipment and staff requirements and make sure that the educational skills of the 
latter are adequate for tneir task, i.e. making it easier for students to learn. 

Tne lack cl an explicit definition of educational objectives makes discussions 
cn prograi^es, teacning methods and evaluation methods difficult and often futile. 

Tne metnods of evaluation (of students, teachers, and methods) will in fact 
depend on the objectives to be achieved. Evaluation (this is ^he third stage) 
consists in being aoie to say to what MEASURABLE EXTENT and HOW the objective laid 
down has been achieved. If care has rot been taken to establish a MEASURABLE 
objective, it will be difficult to rrake a valid evaluation. To choose what type 
of examination to set, for example, without educational objectives makes no more 
sense than trying to choose a measuring instrument without knowing what has to be 
measured. It is only during the last decades that investigators in the field of 
Usting and measurement (docimology) began to work out a solution to the problem 
posed. Here, too, the strength of tradition and emotional reactions are making 
. themselves feit. 



l .For example, the lengtn of medical studies is fixed by administrative 
regulations at 6 years (or 7, 5, etc.): this is the time constant . On the other 
hand, there is hardly any definition of the competence of graduates, and this can 
lead to great variability . 
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The tneme of the workshop will therefore be the health manpower 
training process and willdonsider the three main stages of this process 
{see the Educational Spiral, p. 319): 

- defining relevant educational objectives 

- drawing up ap- effective educational programme 

- implementing a valid evaluation system. 



* 
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SELECTION OF PARTICIPANTS 



(2 months before D day) 



The deadline for applications has now expired and you should convene 
the Committee of Sponsors, They will choose from among >the applicants those 
who correspond most closely to the criteria defined earlier^ (see page 508). 
Your function will be to make sure that the Committee follows these rules 
properly . 

In addition to the number of participants decided on, the Committee 
should select some RESERVES (20-25$ extra) to provide replacements in the 
case of last minute cancellations. 



CONFIRMATION OF PARTICIPANTS 



(45 days before D day) 



It is now time to write to the applicants who have been selected, reminding 
them of the conditions of participation ( full-time attendance obligatory ), the 
place and the dates, asking them to bring Documents 1 and 2 with them (having 
studied them carefully) and sending them Document 3 (on the organization of the 
workshop programme) and the EDUCATIONAL HANDBOOK. 

In point of fact, these last two documents could just as well not be 
distributed^until the start of tne wurkshop since they do not have to be studied 
beforehand. However, it has been found from experience that many participants 
complain at the time of final evaluation that they were not given ALL the 
documentation before the workshop (including those who would not have had time 
to read it). In short, although it is not essential for participants to receive 
Document 3 and the HANDBOOK one month before the workshop, there is no reason 
why they should not "nave them. 

2 

Do not forget that you should have an adeq .ate number of Document J 
"Programme Preparation" reproduced as you nave already had done for Documents 
1 and 2 (page 510). 

Qrder now the necessary number of copies of the Educational Handbook from 
your Regional Office or frcm your own national authorities if they have 
ordered a stock of them. 



It could even be said that if a highly motivated future participant 
has studied the Handbook thoroughly, he has to some extent gone beyond the aims 
of the workshop and has no longer any reason to attend as a participant . He 
could eventually help the assistant organizers. 

2 

Take the opportunity nf having Document 4 "Evaluation of the Workshop 
by the Participants'* reproduce i now for distribution towards the end of the 
workshop (see pa&* ?>3h) . 

259 
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PROGRAMME PREPARATION 



In order lo prepare a working programme for the workshop that is velevant 
to yc uT own needs in the field of education, the following objectives ore 
proposed to you (listed in throe categories). Please select those o? interest 
to you that you wish to reach before the end of this workshop. 

As the duration or this workshop is limited try to be realistic in your 
choice. Some of these objeccives may require only a few minutes* work; 
others may require several hours to allow for study of the documents made 
available to you (see suggested text after; the list of objectives). 

Once you have made your selection decide which objectives are the most 
important to you and rank order them. Fill in the last page and give it to 
the workshop organizer. * 

In the light of what you have- selected, you will be able to draw up a 
working programme. You will be given a list qf the participants who have 
chosen the same objectives as yourself, so as to facilitate the organization 
of small working groups sharing common objectives. 



; 
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THE CONCERT OF EDUCATIONAL OBJECTIVES 



ERIC 



1. Define the following tertns: educational objective; prerequisite level; 

institutional objective; specific objective; cognitive domain; affective 
domain; psychomotor domain. 

2.. List the qualities and sources of an educational objective. 

3. Define the principal tasks of the members of a health team. 

4. List the educational objectives, at institutional level, appropriate to 
the institution in which you teach. /* 

5. Define six specific educational objectives corresponding to a teaching' 
theme with which you are familiar, stating explicitly what you consider 
the student should be capable of "doing 11 at the end of his instruction 
(which he was not capable of doing previously), corresponding to cognitive, 
affect and psychomotor domains. 

6. Critically analyse six specific educational objectives (drafted by another 
participant) and indicate especially if they include all the necessary 
elements (activity, content,, condition, criterion). 

7. Assign the educational objectives tc their appropriate domain; affective, 
cognitive or psychomotor. 

8. Make a list of the possible reactions^ your colleagues in the faculty 
if they had to define specific objectives and suggest strategies to 
overcome their reactions. | 



THE TEACHING/USARNING CONCEPT 



9 # Identify the differences existing between the following concepts: 
education, teaching, learning. 

10. Describe the new trends in the teaching/learning system. 

11 # Enumerate ten conditions which facilitate learning. 

12. Indicate the aims and general methods of teaching. 

13. Identify at least two advantages and two disadvantages for each category 
of teaching/learning activity. 

14. Select a teaching method that will make an educational objective easier- 
to attain. 

15. Draw up an^ organizational diagram representing what you consider workable 
in your establishment for the introduction of an "integrated teaching 
programme" with consideration of constraints and strategies to overcome them. 

16. Identify which role as a teacher you intend to assume in order to facilitate 
the learning of students for which you are responsible. 
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17. Identify the obstacles liable to be encountered during the setting up of a 
competency based curriculum and describe the tactics that may be used to 
overcome them. 

18. Pill in a specification table for a programme. 

19. Define, the concept of relevance with regard to a programme, 

20. Define for a specific objective in a non-cognitive domain what theoretical 
knowledge you consider a student should possess BEFORE starting to work in 
view of that objective (prerequisite level ). 

What theoretical knowledge would be necessary for the student to attain 
that objective # 

* ♦ * # * 



THE EVALUATION CONCEPT 



21 . Draw a diagram showing the relationship between evaluation and the other 
components of the education process. 

22 # Define the principal role of evaluation, 

2?- # Define the aim and purposes of evaluation. 

24. Describe the difference between formative and summative evaluation. 

25. Define the organization (in the form of an organizational diagram) 
and stages of an evaluation system which would be appropriate for 
your institution. Indicate particularly: 

a) the most essential decisions you consider should bo taken on 
the basis of an evaluation, 

b) the aims of the system and sub-systems in regard to decisions 
to be taken and the object of the decisions (teachers, students, 
programme ) . 

26. Identify the obstacles to improvement of an evaluation system and 
the tactics to overcome them. 

27. Define the following terms: validity, reliability, objectivity, 
and describe the relationship which exists between them. 

28. Define the following terms: prerequisite level test, pre-test, 
interval test, comprehensive pre-final test and indicate their 
purpose and the stage at which they are carried out. 

29 # List the good and bad points of a test. 

J0 # Compare the advantages and disadvantages of tests in current use. 

31. Establish an evaluation system relevant to each of the objectives 
(see objective 5) ; questionnaire, written examination ("objective" 
or essay), oral examination, direct observation, etc. Suggest 
alternatives. 

32. Design a system to be used by STUDENTS for evaluation of teaching. 
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33. Set an essay question (open book test) or a set of six short open 
answer questions, indicating the answer (s) called for in each case 
in order to obtain an objective scoring (score sheet). 

34. Draw up three multiple choice questions (MCQ) (in the cognitive domain) 
using the principles set out in the documentation. At least two of 
three objectives must measure an intellectual process superior to level 1, 
"recall", (either at level 2, "interpretation of data", or at level 3> 
"problem-solving") . 

35. Distinguish between a relative and an absolute criteria test. N 

36. Calculate the minimum pass level for a MCQ test. 

37. Determine the scoring criteria and minimum pass level for a mini-test 
(composed of the questions mentioned in objectives 33 and 3*0. 

38. Do an item analysis of a question (index of difficulty and discrimination) 
and interpret the results. 

39. Give the advantages and disadvantages of programmed examination. 

40. Fill in a "specification table" for an examination. 



* * # # * 
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THEORETICAL BACKGROUND THAT WILL HELP YOU 
REACH THE EDUCATIONAL OBJECTIVES OF THE WORKSHOP 



For 

objective 


Study the following 
pages of the 
Educational Handbook 


—f 

For 

objective 


Study the following 
pages of the 
Educational Handbook 


1 . 


6 and 7; 105-109; 
116-119 and Glossary 


21 


304 and 305 


2 


120; 138-139; 1^3 


22 


302-315 


3 


111-<114; 120 


23 ' 


318-321, 




118-131 (and obj. 1 & 3) 


24 


316-317 


5 


1J8-148 (and obj. 1) 




322; 338-340 (and 
obj. 22 and 23) 


- 6 


149 (and obj. 5) 


26 


227; 3^3 (and obj. 25) 


7 


151-166; Glossary 


27 


335-339 


8 


173; 227 (and obj. 1) 


28 


407-410 


9 


108; 207-216; Glossary 


29 


404-406; 412-415 


10 


205-206; 259-267 


30 


416-418 


11 


215 and 216 


31 


324-327 (and obj. 5) 


12 


208-209 


32 


329-333 


13 


219-226 


33 


417; 420-421; 429 


14 


209-210 (and obj. 5 & 13) 


34 


152-155; 428-429; 
435-44Q 


15 


227-244 


35 


457-460 


16 


227-231; 245-251 


* 


458-459 (and obj. 35) 


17 


227-231; 236-253 


37 


452-453 (and obj. 33 
and 3*0 


18 


251-253 


38 


465-473" 


19 


See references in 
Glossary p.6l0 


39 


442-450 


20 


116-117; 138-149; 274 
(and obj. 10) 


40 


253; 454 and 455 
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CIRCLE THE NUMBERS CORRESPONDING TO THE OBJECTIVES 
YOU HAVE CHOSEN AND INDICATE THEIR ORDER OP PRIORITY 



Objective 

• 


Order 
of 

priority 


Objective 


Order 
of 

priority 


Objective 


Order 
of 

priority 


Objective 


Order 
of 
priority 


1 




11 




21 




31 




2 




12 




22 




32 




3 




13 




23 




35 




4 




14 




24 




34 




5 




15 




25 




55 




6 




16 




26 




56 




7 




17 




27 




57 




~8 




18 




28 




58 




9 




19 




29 




39 




10 


l 20 




50 




40 





ADDITIONAL OBJECTIVES 
(optional) 

By the end of the wonts hop I should like to be atrle to: 



Name 

(in block letters) 



Tear out this page and give it to the workshop organizer during the first session. 



ERIC 296 



"53T 



STAFF AND EQUIPMENT NEEDED FOR DOCUMENT 
REPRODUCTION DURING THE WORKSHOP 



(One month before D day) 



It will be extremely useful to be able to call on the services of a typist 
for typing the documents resulting ±rom group work. Participants generally 
wish to have access to the results of their colleagues' work. Equipment for 
fast, good quality reproduction will therefore also need to be available. 

It will also be extremely useful to be able to provide each participant 
(before the end of the first day of work) with a list of the participants who 
have chosen the same objectives as himself as this will facilitate discussion 
between participants (see page 524 document 5) and the formation of working 
groups of participants with the same objectives. 



EQUIPMENT CHECKLIST 



(one month before D da/) 



Note pads (one for each participant + 20#) 
Pencils (one for each participant + 50#) 
Rubbers/erasers (one for each table) 
Pencil sharpeners (one for each table) 
Two-hole punch (1) 
Adhesive tape (2 rolls) 
Stapler (1) 

Waste paper baskets (one for each table) 
Drawing pins (2 packets) 

Cardboard envelope files (2 for each* participant) 

Overhead projectors (2) 

Spare projector lamps (2) 

Electric extension flex (6 metres) 

Electric adapter plugs (2) 

110/220 transformer (check local voltage) 

Transparent cellulose sheets (50) 

Marker crayons for writing on cellulose sheets (12) 

(water soluble) 
Blackboard for flip chart 
Photocopying machine 
Dictionary 



You will find this list useful for checking what you should bring with you 
to the workshop and for making sure the day before the workshop begins that 
EVERYTHING is in place. 



PRESS RELATIONS 



Depending on the local situation, it may be worth deciding to inform the press. 
If so, it will always be best for you to prepare a press release yourself rather than 
leaving this task tc a Journalist, no matter how conscientious he may be. 

If you invite the press to interview the participants (for example when the 
group photograph is being taken) the best time for this will be during the break 
on the last day (see timetable on page 534), This is also a time when ycu may 
invite your superiors, those providing funds and other dignitaries, NEVER choose 
the first day for this purpose , , , as the participants will then be in a state 
of considerable confusionl 
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REVIEW OF THE LIST OF PARTICIPANTS 



(one week before D day) 



In some cases there will be cancellations* As these occur you should get in 
touch with the applicants selected as reserves and so fill the empty places. 
Make sure once again that participants have all the documentation required and 
that they know the date and place of Vie workshop. 



ARRANGEMENT OF THE ROOM 



(2 days before D day) 



The meeting room should be arranged so as to: 

- allow participants to sit at small tables in groups of three to five; 

- allow the use of an overhead projector (two would be preferable in 
order to allow two documents to be compared together). 

The following arrangement is suggested for 15, 25 or 55 participants 
(see page 533). 

During plenary sessions, participants need just to turn to face the organizer. 
Make sure that everyone has a 'good view of the projection screen and the discussion 
leader. 



It has been found from experience that the noise threshold (in group 
discussions) is quite bearable and that this arrangement is preferable to 
separating gi pups in different rooms,. It allows for much more flexibility 
in organizing the sessions. 



Make sure that the room is not too near a source of noise (restaurant, 
school, demolition site, etc*)* 

The less the meeting room looks like a "classroom" the better it will be. 

Make sure that each participant has enough table space on which to lay out 
his documents and see that it is well 3 it. 

* 

The overhead projector does not require a darkened room, but you should still 
make sure the day before the workshop starts at a time when natural daylight is at 
its brightest that the picture it projects is clearly visible. Make sure that 
there is at least one electric power point that works and have an electric extension 
flex and spare projector lamp available. In addition have a stock of transparent 
cellulose sheets and marker crayons ( erasable ) available so that participants, or 
you yourself, may illustrate any remarks that may be made in plenary sessions. 

Use the check- list of the equipment you will need during the workshop at the 
time you are inspecting the room on the day before the meeting starts (page 53l). 
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COORDINATING THE ASSISTANT ORGANIZERS 



(2 days before D.day) 



All assistant organizers should arrive at the site of the workshop without 
fail AT LEAST TWO WHOLE DAYS before the start of the workshop and should have been 
told hew important this coordination period is. 

The principal organizer should make sure that each assistant organizer knows 
what is expected of , him during the workshop and is prepared to carry it out. 

The two days preceding the workshop will therefore be a sort of dress rehearsal 
for what is expected to happen during the workshop. Ea.ch assistant organizer ought 
to give an account in his own words of the part he thinks he can play. 

All organizers will need to be thoroughly familiar with the documentation for 
the workshop and these two days should provide a good opportunity for organizers 
to exchange views and prepare the way for working together smoothly. Several 
informal meetings will probably be useful, one of which should be in the room in 
which the workshop is to be held. 



THE EVENING BEFORE 



In theory all the oarticipants will have arrived and have their hotel rooms. 

Before supper, you arc recommended (if financially possible) to organ.ze a "friendly 

drink" to break the ice and allow participants and assistant organizers to make 
themselves known to each other. This should be as informal as possible. 



D-DAY IS HERE 



Although it is in the nature of a workshop not to have a strict hour-by-hour 
timetable in the case of a mini-workshop some details may be given on the procedure 
it is recommended to follow, at least for the first day. The first hours of the 
first day are of crucial importance. The working atmosphere will change in the 
course of the three days, and roughly speaking the first day will be one of confusion, 
the second one of productive thought and the third one of stunned realization that 
there is much more to learn than appears at first but that it is worth making the 
attempt and the real work is only Just beginning .... 

With regard to the rate at which the subject matter is dealt with in the time 
available, each participant should be left to work at his own pace and according 
to his own svstem of priorities. On the other hand, as the participants form 
themselves into working groups, a "common tempo" could be established as follows: 
It would seem appropriate that Chapter I of the HANDBOOK, which was started on the 
first day should be "finished" by the break on the second day, that Chapter II 
should be' considered during the second session on the second day so as to leave the 
third day for Chapters III and IV. 



TIMETABLE OF WORK 



It is not recommended to arrange more than five hours (two two-and-a-half hour 
sessions) of structured sessions a day. On the other hand, two to three hours (of 
non -structured work) will be needed for individual exercises or for reading documents. 
It would seem from experience that a single morning session lasting four to five 
hours (8 to 10.50; and 11 to 1.00) with a short break (30 minutes) is to be 
preferred to two sessions (morning and afternoon) separated by a lunchbreak. 
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FIRST DAY* FIRST SESSION 
8 to 10.30 a.m. 



1. Opening 

In your opening remarKS you will of course formally welcome the participants , 
thank those who have made the workshop possible, recall the overall aims of the 
workshop (see page 504 ) and situate it in relation to the local teacher training 
situation. Make clear too that the workshop offers each participant a golden 
opportunity for uninterrupted thought on problems that are universally admitted , 
to, be important but are frequently neglected; that no-one there is any cleyerer 
than anyone else; that the workshop belongs to the participants and will^be what 
they make of it; and that the third day will not be the end of th& Workshop but 
rather the start of a long and exciting process. These remarks should not ^taktT 
more than five minutes. 

2. Document clarification 

.Go on without a break to this item. Ask the participants to turn to Document 
No.l .(Working Methods of the Workshop) and go through it page by page and paragraph 
by paragraph asking them if there are any points which they wish CIARIFIED. If 
there are no questions on a paragraph, describe the central theme of the paragraph 
without going into details. If a question is raised, ask whether any other parti- 
cipant, would like to clarify the point concerned. Do not forgot (and remind 
participants where necessary) that tne object o£ the exercise is to clarify yoscure 
points and not to discuss the subject matter in depth. There will be time for such 
discussion throughout the rest of the workshop either in small working groups or, 
occasionally, in plenary session. 

It has been found by experience that there is no point at this stage in a 
clarification session for Document No. 2 (Theme of the workshop). The requisite 
definitions and detailed explanations are dealt with in the exercises designed to 
help achieve the workshop objectives. 

It will now be between 8.45 and 9.15 a.m. and time to pass on to the next item. 

3. Programme preparation 

The (individual) programme of work for each participant will now be prepared. 
Ask participants to turn to Document No. 3 (Programmt preparation) and make sure that 
the text of the first page has been understood. Next, mention that the 40 
objectives listed are the same as those in the EDUCATIONAL HANDBOOK and that tne 
cross-references to the relevant pages of the Handbook have been given (on page 528, 
Document 3) to facilitate selection of objectives. The selection made should be 
indicated by filling in the last page of Document 3 and handing it in to the assistant 
organizers by 10.30 a.m. at the latest. After the break, from the information produced 
in this way, you will post a list of participants who have chosen identical 
objectives so that they may form themselves into groups for Joint work. 

The first plenary session will now be over and the time will be between 9 and 
9.30 a.m. From this point on until the start of the break (10.30 to 11.00) it will 
be the task of the assistant organisers to deal individually with any requests from 
participants. Towards 10.00 a.m. make a rapid tour of the room to see how the 
participants are doing and by about 10.15 a.m. remind them where necessary that they 
have 15 minutes left for handing in the last page of Document No. 3. 
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FIRST BREAK 



(10.50 to 11.00 a.m.) 



This is when the organizers will draw up the lists (on a flip chart) of the 
participants who have selected identical objectives in. order to enable them to 
form groups fcr Joint discussion. Gifted as you are with ubiquity you should 
also make sure that coffee, tea or other drinks, ani rolls and sandwiches will 



be made available to the participants. 
\ . 



FIRST DAY: SECOND SESSION 
U.'OO a.m. to 1 00 p.m. 



By about 10.55 a.m. invite the participants to come back to the room and tell 
. them* they yzy stirt working (on an individual basis to start with) through the list 
of objectives they have selected; that if they wish to find out what they know, 
they ; should now take the pretest (at least on the first chapter of the Handbook); 
that ^hey may call on you if they need any help; that they may subsequently, if 
they so wisn, start exchanging their work and discussing it with other participants, 
whom they may choose from the list given on the flip chart (which you may now have 
reproduced for general distribution); and zhat there will be another plenary 
session at 12.^5 p.m. for the daily evaluation session. Until then take no other 
action at rt from b|ing available to give help if requested. 

At 12.4p p.m. ask for everybody's attention and invite each table to discuss 
(in five minutes) the good and bad aspects of the first morning and ask for a verbal 
report of his discussion from a representative of each table. 

At 12.55 P.ni. you will have a few minutes left to congratulate participants on 
their dedication and keenness and reassure them if they *r-<* feeling completely 
confused about what is happening that things will be bets r tomorrow and even better 
the day after tomorrow. Lastly, invite them to spend the afternoon continuing to 
clarify their ideas by further reading on any points worthy of attention and tell 
them that you will be available for individual consultation. 

It will now be 1.00 p.m. and you may take your leave of the participants and 
wish them a pleasant afternoon.* 



SECOND DAY 



The participants will start organizing tneir work themselves and your task will 
be to help them do this. Start the day with a short clarification session (15 
minutes) which this time it will be worth your while to initiate within the groups 
and finish in plenary session. Next, mention that as on the previous day there will 
be a plenary session at 12.45 p.m, for an evaluation of the day's work and invite 
everyone to set to work. Add that in the event of one (or several) working groups 
forming and wishilig to submit the resuJts of their work to the others, other short 
plenary sessions may be held on request. Make sure that, should this happen, there 
are enough transparent cellulose cheets and marker crayons for participants to make 
their reports using the overhead projector. 

At the end of the session, after evaluation of the day's work, invite participants 
to draw up for the following day a list of the professional objectives they wish to 
reach during tjie next 12 months in the light of what they have "learnt" during the 
workshop, and^sk them to fill in the evaluation questionnaire (Document No. 4) and return 
cages 539-544 to you at 6 p.m. (page 546 will tell you how to analyse the results 
c p the ques' ionnaire). 

1 Another scenario may be imagined in Vhich bv 8 30 a.m. the Pgrticigants^iave 
S pi£^ 
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DOCUMENT NO. 



EVALUATION OF THE WORKSHOP 
BY THE PARTICIPANTS 
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QUESTIONNAIRE FOR EVALUATION OF THE WORKSHOP BY THE PARTICIPANTS 

Instructions for questions 1-r- 35 

Use the following code to indicate the exten* to which you agree or disagiee 
with each of the statements made below: 

Code 1. Strongly disagree 
2. Disagree 

4. Agree 

5. Agree strongly 

The difference between 1 and 2 or between 4 and 5 is one of degree only. 



Example: If you want to express your complete disagreement with the 
statement, circle the figure 1 as follows: 2 4 5 



Please feel free to make any comments you think necessary (making reference 
to the number uf the question) ir. the space reserved for the purpose on the last 
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I # Aspects relating to the 'planning of the workshop 

Q.l I was given sufficient information on the 
aims and methods of the workshop before my 
arrival 

Q,2 The planning of the workshop reflected the 

educational principles that were discussed 

there f 
Q.3 It was clearly explained to me at the start 

how I was to "choose ray objectives for the 

workshop 

Q # 4 I feel that I participated in the prepara- 
tion of the pi ^gramme in a satisfactory 
manner 

Q.5 The goals of the workshop appeared to me 
to be of immediate interest for my pro- 
fessional activities 
Q # 6 It was clear to me from the start of the 
workshop that I was expected to play an 
ictive part in it 



a^ci 



Do not write 
in this column 



12 4 5 



12 4 5 



12 4 5 



12 4 5 



12 4 5 



12 4 5 



r 



c 



CD 



CD 



CD 



CD. 



II. Aspects relating to the relevance and utility of the working methods 

Q # 7 I found the documentation provided of 

an acceptable quality 12 4 5 

Q.8 Enough documentation was provided to allov; 
me to take an active part in the discussion 
of the subjects concerned 12 4 5 



CD 



CD 
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Do not write 
in this column 



Q.9 The information given by the EDUCATIONAL 
t HANDBOOK helped me to reach the. objectives 
I had chosen for the 'workshop 

Q.10 The working methods used during the work- 
shop encouraged me to take an active part 
in it 

Q.ll I have already during the workshop begun to 

put new knowledge into practice 
Q.12 Spending as much time as possible on 

individual* work during the workshop helped 

"ie to learn 
Q.13 During the workshop I was given the 

opportunity to work at my own pace 



12 4 5 



12 4 5 



12 4 5 



12 4 5 



12 4 5 



CD 



CD 



CD 



CD 



CD 



III. Aspects relating to the way the workshop was run and to the 

attitude of the organizers 
Q.14 The organizers displayed a 

satisfactory open-raindedness 12 4 5 

Q.15 The general atmosphere of the workshop 

was conducive to sei'ious work 12 4 5 

Q.16 The organizers gave me opportunity 

for critical comment 12 4 5 

Q.17 The organizers made use of any critical 

comments I made during the workshop 12 4 5 

Q.18 The organizers made every effort to help 

me reach my objectives for the workshop 12 4 5 



CD 



CD 



CD 



CD 
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Q.19 The way the workshop was conducted was in 
line with the educational principles it 
discussed 

0.20 'I consider that enough time was given for 
individual or group discussions .with 
the organizers " r * 

Q.21 Enough^ time was devoted to clarifying the 
documents 

Q.22 Enough time was given to smail group 
discussion 

Q.23 Enough time was given for practical 
exercises 



12 4 5 



12 4 5 

12 4 5 

12 4-5 

12 4 5 

12 4 5 



24 Enough time was given for individual work 
Q.25 Enough time was given for the presentation of 

f> work inplenary session 12 4 5 

Q.26 The attitude of the organizers was 

conducive to "free learning" 12 4 5 



Do not write t 
in this column 



CD 



CD 



C 
C 

c 
c 



3 

3 



CD 
CD 



IV. Aspects relating to the benefits gained by the participants 
Q.27 The workshop helped me to improve my 

knowledge of education theory 12 4 5 

Q.28 The workshop helped me to develop a 

favourable attitude towards the systematic 

4 

approach to educational problems 12 4 5 



CD 
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Do not write 
in this column 



Q.29 The workshop has encouraged me to put the 
knowledge I have gained into practice 

after the workshop "is over 12 4 5 

Q^O The workshop will help me to encourage ray 

colleagues to learn and make use of new 
\ educational methods 12 4 5 

\ 

Q.31 \ The workshop has increased my confidence 
in my ability to achieve my personal 
objectives in the medium-term (in one year's 
time) 12 4 5 



QD 
CD 



CD 



V. Aspects relating to evaluation of the workshop 
Q.32 I felt t^at the pre-tast and the follow-up test 

helped me to make a useful assessment of the 

knowledge I gained 1 2 4,5 

Q.33 The pre-test was a useful exercise and showed 

the advantages of this technique 12 4 5 

Q.34 The practical exercises showed the usefulness 

of "feedback 11 during the learning process 1 2 4 5 

Q.35 I found the daily evaluat n sessions useful 12 4 5 



CD 
CD 



C 
C 
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With regard- to the planning of the workshop, its method of work and the 
attitude of the organizers, note below and give actual examples of: 



(a) The factor? which impressed you most favourably 



\ 



(b) The factors that impressed you least favourably 
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Further comments and suggestions 



/ 
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VI # Professional objectives 



1, Write down the objectives you hope to achieve during the year 
following this workshop, 

2, Write down, for each of these objectives, a working timetaole to 
enable you to reach them. 



(continue on the other side) 
NAME 

Keep a copy of this page and give the original to one of the workshop organizers. 
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Note for the Organizer on what to do with the answers 
to this Questionnaire 



A very simple analysis may be carried out as follows. 

Take an uncompleted questionnaire and mark beside each question the answers 
given by eabh participant. For example, for 50 participants, the answers to 
•question 6 might be: 



Q.6 It was clear to me from ,lhe start of 
the workshop that I waa expected to 
play an active part in it 1 



2 4 5 



In other words, two participants c^.isidered that they did not understand from the 
start that they were expected tc play an active part in che workshop while the 
28 others understood this. Multiplying the number of answers by the corresponding 
coefficient gives a total of (2 x 2 + 10 x 4 * 18 x 5) - (* + ^ + 90) - W> 

. The ''satisfaction index" is calculated by multiplying this number by 20 
(i.e. 100 divided by the maximum coefficient 5) and dividing it by the number of 
participants, in this case j50. This gives 



\y\ x 20 

30 



2680 
50 



= 89.5^ 



You are tnen recommended to make a note of any questions with a "satisfaction index" 
BELOW 60# . If there are none, identify the five questions with the lowest 
"satisfaction index" and then the five questions with the highest "satisfaction index" 
Let the participants have these results at t^e final evaluation session on the last 
day of the workshop. 



The satisfaction index is calculated in such a way that 
satisfaction" is ICO + 20 _ ^ 



average 
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THIRD DAY 



(don't forget to have a group photograph taken during the break) 



After a short period for clarification, set aside the first session for work 
on Chapters III and IV o? the Handbook. It is very important to keep the second 
session of this last day for finalizing the individual professional objectives to 
be reached during the next 12 months. (See Document No. 1, paragraph 8.4). 
These should be the subject of exchanges of views between participants and of group 
discussion before being handed to the organizers. The last half -hour of the last 
day should be given over to an evaluation session ending with a few closing remarks 
dealing mainly with the FUTURE. 



LETTER OF THANKS 



(15 days after the workshop) 



It is no., time to thank everybody who has he]ped you, including those who have 
provided funds for the workshop, the members of the Committee of Sponsors and the 
assistant organizers, if an^. Infonn them in the letter that they will shortly 
be sent the report on the workshop. 



REPORT ON THE WORKSHOP 



(15 days after D day) 



Even if the purpose is only to inform those who have provided the funds for 
the workshop or to help participants to inform other colleagues , it is a useful 
exercise to prepare a report during the week following the workshop. 

The report will contain: 

- an INTR0DUCTI01J giving a brief description of the local context which led 
to the organization of the workshop 

- the GENERAL AIMS of the workshop 

- a description of the PROGRESS of the workshop describing how participants 
adapted themselves to the working methods 

- some selected samples o'f the results of individual or group work 

- the list of documents us3d during the workshop 

- the results of immediate evaluation (including an analysis of Document No, 4) 

- the list of participants (with their addresses) 



LONG-TERM EVALUATION 



(at the latest one year after D day) 



Longterm evaluation is essential since it is the only way which will give 
some measure of the ACTUAL impact of a workshop. 



It will be based on an assessment of the extent to which each participant has 
reached, failed to reach or exceeded all or some of the PROFESSIONAL OBJECTIVES he 
set himself at the end of the workshop (see page 545). An assessment of this kind 
may £e made by means of a questionnaire, by individual interviews or preferably 
by bringing all participants together again for a one-day meeting. A report 
summing up the results of long-term evaluation is well worth preparing. 
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AFTERWARDS 



Do not succomb to the illusion that everything is now going to change in 
the institution in which you work. Do not think that from one day to the next 
conservative elements will turn into reformers, passive elements into active 
ones or opponents into supporters. 

Firstly, if you manage to persuade at least 20# of those taking part in the 
workshop to make a lasting and visible change in their teaching habits during the 
ensuing year, you may consider the workshop to have been a success. This^gill 
not be the case if all you do is record how many participants expressed satis- 
faction during the evaluation on the last day of the workshop (even if the 
figure is 100%). 

Secondly, your work is not yet over. Do not xet the seed you have planted 
with so much effort wither away. Continue to stimulate your colleagues in 
responsible positions in your teaching institutions to see that those participants 
who improve their teaching methods are "rewarded" in some way. 

If this is impossible, do something about it yourself - congratulate them - 

this will always be appreciated. 



A WORKSHOP - — - 

DON'T THINK IT WILL CHANGE NOTHING : 

DON'T BELIEVE IT WILL CHANGE EVERYTHING 
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THIS CHAPTER HAS BEEN PREPARED FOR TEACHERS WHO HAVE ATTENDED AT 
LEAST ONE EDUCATIONAL WORKSHOP AND THUS KNOW HOW SUCH WORKSHOPS 
OPERATE AND ARE FAMILIAR WITH THE EDUCATIONAL HANDB9OK, 

It is offered to you for use in a wide range of 
local contexts. This means that any user will always have to remember 
to make allowance for the cultural, educational and occupational backgrounds 
of participants. Modifications may therefore be contemplated, worked out, 
implemented and, above all, evaluated. However, a user who has had no 
^ special training in educational science is not advised to introduce 

significant changes right away. It is better to use the proposed system 
as it stands and be prepared to amend it in the light of experience, inter 
alia as a result of "feedback" from participants* 

* 
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This glossary of technical terras used in educational and psychological measurement is 
intended primarily for persons with limited training in test and measurement (docimology) , 
rather .than for specialists. The tSrms defined are those most commonly found in test 
manuals and simple research reports. In the definitions, niceties of usage have some- 
times been sacrificed for the sake of brevity, and, it is hoped, clarity. 

The terms and definitions included are purely ad hoc , for the purposes of this 
glossary alone, and they represent only the views of the authors and do not necessarily 
have any validity outside the context of the handbook. There is not complete uniformity 
among writers in the measurement field with respect to the usage of certain, technical terms 
in cases of varying usage, either these variations are noted or the definition offered is 
xhe one that the writer judges to represent the "best" usage. 



* Adapted from Roger T. Lennon, "Test Department" Harcourt, Brace and World Inc. 
Revised by J. -J. GUILBERT (WHO); G« and L. MASSE, ENSP, Rennes (France); B. PISSARRO, 
University of Paris (Prance) and J. C. Chencerel, University of Neuch&tel (Switzerland), 
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Absolute criteria test -(see p. 457 to 460.) 

Affective - Concerning the feelings; affectivity is the combination of acts and tendencies 
which express the feelings felt towards others; it is situated primarily on the interhuman 
relationships level. It is often employed to qualify attitudes. The word "affect 1 ' is a 
neologism used by psychologists to designate an affective state (pp, 102, 106, 156-158 162- 
166 : , 324, 326-327)1 



Aptitude - A combination of abilities 'and other characteristics, whether inburn or acquired, 
known or believed to be indicative of an individual's ability to learn in some particular area. 
Thus, musical aptitude would refer broadly to that combination of physical and mental charac- 
teristics, motivational factors and conceivably other characteristics, which is conducive to 
acquiring proficiency in the musical field. Some exclude motivational factors, from the 
concept of "aptitude", but the more comprehensive use seems preferable. The layman may think 
of "aptitude" as referring only to some inborn capacity ; the term is no longer so restricted 
in its psychological or measurement usage. 

Arithmetic mean (one of the central tendency values of a distribution) - The arithmetic mean 
is the sum of the observations divided by the number of observations (p. 462). 

Asymmetry - "Characteristic of a distribution where the value* are not equally distributed on 
either side of the mean. In this case: (1) the three central tendency characteristics do 
not coincide, (2) the standard deviation has no longer any geometric significance in describing 
the dispersion of the distribution, which is then better expressed by the quantiles (quartiles, 
deciles, percentiles)" (p. 478). 

Attitude - Derived from the latin "actitudo". which indicates a disposition. By analogy, the 
internal disposition whi r h is shown (internally) in behaviour with respect to persons, events, 
opinions or theories, it is a relatively constant disposition of feeling toward? someone or 
something (pp. 106, 114, 116, 258-265, 326-332). 



B 

Battery of tests * A group of several tests standardized on the same population , so that 
results on the several tests are comparable. Sometimes this term is wrongly applied to any 
group of tests administered together, even if they are not standardized on the same subjects. 

Behaviour - The total reactions (of an individual) accessible to external observation. 
Thought and understanding are implicit behaviours which are observable not directly, but 
solely by inference from other observable behaviours (pp. 2, 7, 106, 108 116 163 258-265 
'314, 323). 

Bias - "Bias is the result of an incorrect procedure for selecting a sample, so that the 
sample is not representative of the population from which it is taken." 

Bit - Abbreviation of "binary digit". In cybernetics, the basic unit of information. 

c 

Coefficient of correlation (r ) - A measure of the degree of relationship, or "going- 
togetherness" between two sets of measures for the same group of individuals. IVo coeffi- 
cients of correlation are used in psychology, firstly the Pearson (r), developed at the 
beginning of the century by the British statistician Karl Pearson, and, secondly, Spearman f s 
correlation coefficient (r s ). The first is more powerful, but the data observed must be 
normally distributed and their variances must be homogeneous. Furthermore, the calculation 
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t ikes longer unless a computer is available. The second is less powerful but does not assume^ 
aiy preliminary condition of the data and is easier to calculate. Unless Otherwise specified, 
'Correlation" usually means "coefficient of correlation", which ranges from 0 (showing complete 
absence of relationship) to 1 (denoting perfect correspondence). It may be either posit 'e or 
negative '*so th*u it ranges from -1 to +1 (see pp. 461-463). 

Coefficient of reliability - The coefficient of correlation between two forms of a test, 
between scores on repeated administration of the same test, or between halves of a test, 
properly corrected. These three coefficients measure somewhat different aspects of reliahi~" 
lity. See, for example, the Kuder-Richardson formula (see pp« 461-463). 

Cognitive s - Cognitive field: the field of knowledge. The "cognitive faculty is the act of 
knowing" (see pp. 2, 7, 106, 108, 152-155, 163, 314, 323-324). 

Collective test - A test that can be administered to a certain number of persons at the same 
time (see individual test). 

Competence - The ability required to assume certain functions (see pp. 105, 106, 206, 233-235). 

Completion item - A test question calling for the completion of a phrase, etc., from which one 
or more parts have been- omitted (see page 423). 

Correct ioa for guessing - Reduction in score to allow for guesses, sometimes applied in 
scoring questions of the true- false or multiple-choice type. This device is intended to 
discourage guessing and to yield more accurately rankings of examinees in terms of their true 
knowledge (page 474). 

Correlation -.Relationship or "going-togetherness", between two scores or measures. The 
existence of a high correlation between two variables does not necessarily indicate that one 
has any causal influence on the other (see coefficient of correlation) (see p. 461-463). 

Criterion - A standard by which a test may be judged or evaluated; a set of scores, readings, 
etc., that a test is designed to predict or to correlate with (see validity) (see pp. 146, 148, 
165-166, 311-312, 314, 322, 457-460.) 

Criterion referenced test -^see pp. 457-460.) 
Critical incident technique - (see pp. 111-112). 

Cybernetics - The science and technique of information processing; it enables problems to be 
rapidly solved and implementation to be automatically set off. In education, this corresponds 
to teaching organized in such a way that the student acquires the knowledge and "know-how" 
called for by the educational objectives by means of systematically established exercises 
(programme teachingor teaching machines , for example) and is informed about his progress (see p. 105/ 
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Decile - There are nine deciles and they are those values of the variable that divide the 

distribution studied into 10 equal parts; the fifth decile is the median (see this word); 

in scoring the first decile denotes the score below which 10% of the scores fall (see p. 478), 

Deviation - The amount by which a score differs from some reference value, such as the mean, 
the norm or the score on some other test. 

Diagnostic Test - In education, a test used to "diagnose", that is to locate, specific areas 
of weakness or strength and to determine the nature of the weaknesses or deficiencies; it 
yields measures of the components or sub-parts of some larger body of information or skill. 
Diagnostic achievement tests are most commonly prepared for assessing skills (see p. 316)". 

Difficulty index - The percentage of some specified group of students who answer an item 
correctly (see pp. 465, 469-473). 
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Discrimination index - The discriminating power of an item (see p p . 465 f 469-473). 

Distractor - Any of the incorrect choices in a multiple-choice question (see p. 482)* 

Distribution (frequency distribution) - A tabulation of scores from high to low or low to high, 
showing the number of individuals that obtain each score or fill in each score interval (seep* 467 )♦ 

Doc jmo logy - Neologism suggested by French psychologist H. Pieron to designate the scientific 
study of test methods (see pages 107, 403)* 

E 

Education - (see pp. 108-309). 
Epidemiological studie s - (see pp. 112-113). 

Equivalent form - Any of two or more forms of a test that are closely parallel with respect to 
the nature of the content and the difficulty of the items included, and that will yield very 
similar average scores and measuies of variability for a given group (see p. 461) . 

Error of measurement - See standard error. 
Ethics (professional ) - (see p. 259). 

Explicit - Stated in a sufficiently clear and precise manner, so as to leave nc doubt. ANT: 
implicit, confused, equivocal, ambiguous, obscure (see pp. 108, 147). 

Extrapolation - In general, any process t>f estimating values of a function beyond the range of 
available data. As applied to test norms, the process of extending a norm line beyond the 
limits of actually obtained data, in oruer to permit interpretation of extreme scores. This 
extension may be done mathematically by fitting a curve to the obtained dat£ or, as is more 
common,. by less rigorous methods, usually graphic. Considerable judgement on the test maker's 
part enters into any extrapolation process, which means that extrapolated norm values are 
likely to oe to some extent arbitrary. 

F 

Factor -,In mental measurement, a hypothetical trait, ability, or component of ability, that 
underlies and influences performance on two or more tests, and hence causes scores on the test 
to be correlated. The term "factor" strictly refers to a theoretical variable, derived by a 
process of factor analysis, from a table of interrelationships among tests; but it is also 
commonly used to denote the psychological interpretation given to the variable - i.e. the 
mental trait assumed to be represented by the variable, as verbal ability, numerical ability, 
etc. 

Factor analysis - A ^roup of statistical methods for analysing the intercorrelations among a 
set of variables (for example, test scores). Factor analysis uncovers factors which explain 
the common characteristics of and the differences between the examinees and the questions. 
Factor analysis has contributed to the understanding of the organization or components of 
intelligence, aptitudes and personality, and it has pointed the way to the development of 
tests of the several components. "Factor" has no causal meaning here, but is used in its 
mathematical sense and is not intended to^ express a causal relationship. 

Feedback - As informative reaction to the message of teaching. It is useful , not only as a means 
of checking, but also as ameans of increasing the students learning activicy (see pp. 320-321, 329K 



G 

G rade norm - The average score obtained by students of a given level (see p. 466). 
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H 

Halo effect - An interference effect resulting from the idea the examiner has already formed 
of a student (previous results) on the basis of his test results (see page 453). 

I 

Individual test - A test that can be administered to only one person at a time (see collective 
test) . 

Integration - To join parts together to form a consistent whole. Coordination of different 
learning/ teaching activities with a view to the harmonious functioning of the educational 
process and more effective training of personnel (see pp. 203, 229, 239-250). 

Interpolation - In general, any process of estimating intermediate values between two known 
points. As applied to test norms, it refers to the procedure used in assigning interpreted 
values to scores between successive limits obtained in the standardization process. In 
reading norm tables it is necessary at times to interpolate to obtain a norm value for a score 
between scores given in the table (see p. 478). 

Item - A single question Or exercise in a test (see p. 465). 

Item analysis - The process of evaluating single test items by any of several methods, in 
particular by determining its difficulty index, discrimination index and o£ten its correlation 
with some selected criteria (see pp. 465-473). 

K 

Kuder-Richardson formulas - Formulas for estimating the reliability of a test from information 
about the individual items in the test, or from the mean score, standard deviation, and number 
of items in the test. Because the Kuder-Richardson formulas permit estimation of reliability 
from a single administration of a test, without it being necessary to divide v the test into 
halves, their use has become commjn in test development. The Kuder-Richardson formulas are 
not appropriate for estimating the reliability of speed tests (see page 462)". 

L 

Learning - (see pp. 106, 117, 151, 201-206, 212-216, 229, 232-235, 258-265, 313.) 

M 

Machine- scorable (ma, nine-scored) test - A test which may be scored by means of a machine. 
In taking tests that are to be scored by machine the examinee records his answers on separate 
answer sheets. The machine rapidly distinguishes between right and wrong answers and can 
combine groups of responses in order to yield total or part scores, weighted scores, or 
corrected scores. 

Matching item - Type of multiple-choice item calling for the correct association of data given 
in one list with those given in a second list. There are several types: simple matching, 
excluded term matching, causal relationship, quantitative comparison, relative variation (see 
pp. 431-432). 
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Median (one. of the central tendency values in a distribution) - The median is that value of a 
variable which divides the series into two equal parts when the data are arranged in increasing 
or decreasing order . The median corresponds to the second quaitile, the fifth decile and the 
fiftieth percentile. For example, in a gtoup of scores, half of the group lies above the 
median and the other half below (see p, 478). 

Mode (one of the central tendency values of a distribution) - The mode is that value of a 
variable which occurs most frequently in a distribution (see p, 478). 

Multiple response items - Type of multiple-choice item in which two or more of the given choice 
may be correct and where there is generally a code permitting only one answer to be given on 
the answer sheet (see p. 430-431). 

Multiple- choice question (MCQ ) - A test item in which the examinee's task is to choose the 
correct or best answer from several given answers or options (distractors ) (see p. 314-315, 
418-419, 428-440). 

N 

N - The symbol coiunonly used to represent the number of cases in a distribution, study, etc. 

Norm referenced test -(see p. 457-460.) y - 

Normal or GAUSS distribution - A distribution of scores or measures that in graphic form has a 
bell-shaped appearance This distribution, termed the normal distribution, which is an 
incomplete and in any case misleading designation in the human sciences, particularly in 
psychology, is a theoretical distribution to which a certain number of observed distributions 
conform (the figure on page 478 is an example). In a normal distribution the values are 
symmetrically .distributed about the mean; the mean, mode and median are merged because of this 
symmetry. There is. a concentration of values around the mean, with a decrease in frequency 
the further one departs from the mean. 

A certain number of observed distributions follow the normal distribution law, and 
certain observed distributions which are not normal can sometimes be made normal distributions 
by transforming the variable , which facilitates certain statistical treatments (variance 
analysis and correlation calculations in particular). 

Moreover, the normal distribution is useful in sampling judgement, for the sampling 
.distributions of various parameters, the mean, for example, or the standard deviation, follow 
the normal distribution law if the sample si2e is adequate, However, care must be taken not 
to confuse the observed distributions with a sample or a population and the sampling distribu - 
tions , which are theoretical distributions making possible to predict sampling fluctuations of 
relative frequencies, or of parameters such as the mean or standard deviation (see p. 478). 

Norms - Statistics that describe the performance of a group in given tests, e.g. of students 
of various' grades in a group in a standardized test, Norms are often representative of some 
larger population as of students and the country as a whole. They are not to be regarded as 
standards, or as desirable levels of attainment, Grade, age and percentile are the most 
common types of norm (see pp. 457, > 478). 

0 

Objective test - A test in the scoring of which there is no possibility of differences of 
opinion among scorers as to whether responses are to be scored right or wrong. It is con- 
trasted with tests such as the traditional essa> examination to which different scorers may 
assign different scores, ratings or grades, e.g. MCQ, programmed examination (see pp. 417-418- 
426-450). 

Oral examination (see p. 4l6) 
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Percentile - There are 99 percentiles and they are those values of the variable which divide 
the distribution series into 100 equal parts, the fiftieth percentile is> the median (see this 
word). To each percentile there corresponds a valup of the variable such that the percentage 
of values observed below that value is indicated b^ the rank of the percentile: thus, in 
scoring the fifteenth percentile demotes the score below which 15% of the scores fall" (seep.^ 

Percentile rank - Percentage of scores in distribution equal to or lower than the score 
corresponding to the given rank (see p. 478). 

Performanc e (or control) test - A test which measures to what extent a student has "achieved" 
something - learned certain data or mastered certain techniques following a given course of 
teaching, a practical test requiring motor or manual response on the part of the examinee. 
Generally, but not always, it involves manipulation of concrete equipment or materials. The 
term is also used in another sense to denote a test that is actually a work-sample. In this 
sense it may include paper and pencil tests, such as, for example, a test in statistical 
calculation where no materials other than paper and pencil are required, but where the test 
response is identical with the behaviour about which information is desired (see p. 108, 147-1 

Personality test - A test intended to measure one or more of the non- intellectual aspects of 
an individual's mental or psychological make-up. Personality tests include the so-called 
personality inventories or adjustment inventories (e.g. Heston Personal Adjustment Inventory, 
Bernreuter Personality Inventory, Bell Adjustment Inventory) which seek to measure a person's 
status on character traits (sociability, introversion, etc.) by means of self-descriptive 
responses to. a series of questions; rating scales (e.g. Haggerty-Olson W^ckman Behaviour * 
Rating Schedules) which call for rating by oneself or another of the extent to which an 
individual possesses certain characteristics; situation tests in which the individual's 
behaviour in simulated life-like situations is observed by one or more judges, and evaluated 
with reference to various personality traits; and opinion or attitude inventories 
(e.g. Allport-Vernon Study of Values). The results are particularly difficult to interpret. 

Practice effect - The influence of previous experience with a test or a later administration 
of the same test or a similar test. Usually there is an increase in the score on the second 
testing, attributed to increased familiarity with the directions, kinds of question, etc. 
Practice effect is greater when the interval between testings is short, when tne materials in 
the two tests are very similar, and when the * i.tial test is a relatively novel experience for 
the subjects (see p. 330). 

Prerequisite level test - In education, a test that measures the extent to which an individual 
has acquired a certain level ojj competence needed to undertake some new learning activity 
successfully. For example, a prerequisite level test in pharmacology shows the extent to 
which a student has reached a level whereby he may profitably begin a course on therapeutics. 
Also called readiness test (see pp. 138, 407-409). 

Pre- test - Test which shows to what extent a student has acquired certain knowledge before the 
commencement of the corresponding teaching. 

Prognosis (prognostic) test - A test used to predict future success or failure in a specific 
field or subject (see pp. 9, 407-409). 

c 

Programme - Plan laid down in advance fixing the order or the timetable of a group of educa- 
tional activities (see pp. 6-7, 105-107, 114-117, 201-266). 

Programmed learning - Method of instruction consisting of a system of graduated questions with 
an immediate check of the answer, the right answer is needed in order to answer the following 
question correctly. Such teaching can be automated by using, for example, teaching machines 
or computers (see p. 221). 



Projective technique - A method of personality stu*dy in which the subject responds as he 
chooses to a series of stimuli such as ink-blots, pictures, unfinished sentences, etc. So 
called because of the assumption that under this free response condition the subject "projects" 
into his responses manifestations of personality characteristics and organization that can be 
interpreted to yield a description of his basic personality structure. The Rorschach (ink- 
blot) Technique and the Murray Thematic Apperception Test are the most commonly used projec- 
tive methods. 

Psychomotor - Referring simultaneously to sensibility, sensation and raotricity. Synonym; 
sensory-motor (see pp. 102, 106, 159-165, 324). 

Q 

Quanti le (parameter dispersion of data with range and standard deviation) - The mam quantiles 
are the" quart iles , the deciles and the percentiles (see p. 478). 

Quartile - One of three points which represent values of the variable dividing the series 
studied into four equal groups. The second quartile is the median (see this word) in a group 
of scores, the first quartile represents that score below which lie 25% of the scores (see p. 
478). 

R 

£ - See coefficient of correlation. 

Random sample - A sample of the members of a population drawn in such a way that every member 
of the population has an equal chance of being included. This method precludes any bias or 
selection. The purpose of using the method is, of course, to obtain a fairly "representative" 
sample of the total population, so that the sample findings may be generalized to the whole 
population. Random sampling has also the following advantage; in accordance with the 
calculus of proDabilities , formulae are, available for predicting the relative frequencies or 
parameters, such as the mean or standard deviation of the sample, from the true values of the 
relative frequencies or parameters in the total population, conversely, the limits within 
which are very probabl> situated the relative frequencies or parameters of the population from 
which the sample is drawn can be estimated from the relative frequencies^ or parameters observed 
with the sample. There are various methods, in particular random number tables , fcr drawing 
such a sample in practice. 

Range - The difference between lowest and highest scores obtained in a test administered to a 
given group (see p. 467). ? 

Rank- ordering - The process of arranging uie students in order of merit as determined by scores 
obtained in one or more tests (see p. 467). 

Raw score - The first quantitative r.oult obtained in scoring a test (for example , the number of right 
answers , number of errors , oi .»i „i la i , direct , unconverted, uninterpreted measures) (see pp. 466-467) . 

Recall item - An item that requires the examinee to supply the correct answer from memory. 
This item differs from the recognition item in which he need only identify the correct answer. 
For example, "Claude Bernard published the 'Introduction to the Study of Experimental Medicine 1 
in . . .?" is a recall item, whereas "Claude Bernard published the 'Introduction to the Study 
of Experimental Medicine 1 in A) 1785, B) 1815, C) 1865, D) 1905" is a recognition item in the 
form of a multiple-choice question (see p. 423). 

Recognition item - An item requiring the examinee to recognize or select the correct answer 
from among two or more given answers, e.g. in a multiple-choice question (see that term). 
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Relative criteria test (see p. 457-460) 

Relevance - (see pp. 6-7, 105-114, 123, 138-139, 143, 173-174, 219, 232-235, 253, 302, 315, 335-339, 
347, 404, 452, 485). 

Reliability - The extent to which a test is consistent in measuring whatever it has to measure; 
dependability, stability, and relative freedom from errors of measurement. Reliability is 
usually estimated by some form of correlation coefficient (see reliability coefficient, 
standard error of measurement ). (see pp. 302, 312, 334-339, 461-463). 

Representative sample - A sample that corresponds to or matches the population of which it is 
a sample with respect to characteristics important for the purposes of the investigation. 



s 

S^ale - Continuous series of programmed values in which data are situated in order to assess 
them with respect to a whole (see pp. 326-332, 452-453). 

Sensory-motor - See psychomotor. 

Sociometry - Measurement of the interpersonal relationships prevailing among the members of a 
group. By means of sociome,tric devices, e.g. the sociogram, an attempt is made to discover 
the patterns of choice and rejection among the individuals making up the group (by identifying 
those chosen most often as friends or leaders, and those rejected by others), to see how the 
group subdivides, etc. 

Spearman- Brown formula - A formula giving the relationship between the reliability of a test 
and its length. The formula permits estimation of the reliability of a test lengthened or 
shortened by any amount, from the known reliability of a test of specified length. Its most 
common application is in the estimation of reliability of an entire test from the correlation 
between the two halves of the test (split-half reliability) (see page 462). 

Specification tables -(see pages 106, 203, 255-257, 454-455). 

Stanine - One of the steps in a nine-point scale of normalized standard scores. The Stanine 
scale (short for standard-nine) has values ranging from 1 to 9 with a mgan of 5 and a standard 
deviation of 2 (see p. 478) 

Standard deviation - A measure of the dispersion of a set of data. The more the data cluster 
around the mean, the smaller the standard deviation. The standard deviation of an observed 
distribution is useful in particular as a geometrical measure of the dispersion when the 
distribution is symmetrical and, more especially, *.hen the distribution follows the normal 
distribution law (page ^78). In the latter case, the proportion of observations contained in 
a given interval, around the mean, expressed in "standard deviation" units, is mathematically 
defined, in particular, 68% of observations are found within an interval of one standard 
deviation on either siUe of the mean, 95X within an interval of two standard deviations, and 
more than 99% (99.747* to be precise) within an interval of three standard deviations. If the 
distribution is asymmetrical , the standard deviation is still a valuable measure of dispersion 
but is no longer of geometrical value for determining these intervals. On the other hand, the 
variance (square of the standard deviation) is always used in this case to make a certain 
num. ar of estimates concerning the form and spread of the sampling distribution (and no ^nger 
of the observed distribution) (see pp. 461-463, 478). 

Standard error - The standard error is the standard deviation of a sampling distribution . 
The use of the standard error is very important when the sampling distribution follows the 
normal distribution law (i.e. when the sample is drawn at random and is sufficiently large: 
for example, n > 30 in the case of the sampling distribution of a mean) (see pp. 461-463). 
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Standard score - A general term referring to any of a variety of corrected scores, in terms of 
which raw scores may be expressed for reason of Cvnvci.ence , comparability, ease of interpreta- 
tion, etc. ' 

The simplest type of standard score is that which expresses the deviation of an 
individual's raw score from the average score of his group in relation to the standard 
deviation of the scores of the group. Thus: 

Cfo^^ov^ / n _ raw score Cx) - mean (M) 

Standard score (z) = — - — - — — — : : ; 7—7 

standard deviation (SD) 

Standard scores having the s*me mean or standard deviation may be obtained from this ratio. 
Such standard scores do not affect the relative standing of the individuals in the group nor 
change the shape of the original distribution (see conversion chart, page 478). 



Task analysis (see p. 112). 
Teaching (see pp. 205-210). 

Taxonomy - (see pp. 151-163, 314-315) 



v 

Validity -(see pp. 302,^312, 329, 335-339). 

w 

Workshop - (see pp. 501-549). 
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Note : The questions are set on pages 167 to 170, 269 to 271, 344 to 346 and 479 
to 482. 
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to the Post Test (page 517) 
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Recheck on pages 172,273, 348, 484. 
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The list of references has been kept intentionally short and is divided into the following 



1. Health needs and resources, organization of health services, analysis of tasks 
and functions. 

2. Psychology of education, medical (Health sciences) education (general). 

3. Educational objectives* 

4. Curriculum planning and teaching techniques. 

5. Multidisciplinary student laboratories. 

6. Evaluation and measurement (docimology) . 

7. Evaluation by students. 

If you would like a more detailed and fuller list of references, you may apply either to 
the WHO representative in your country, to the WHO Regional Office, or directly to: 



categories: 



.1 



Educational Planning, 

Division of Health Manpower Development, 
World Health Organization, 
1211 Geneva 27 



> 



Note, however, that many of these references concern several fields. 
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